[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:56 dist-test-slave-s0b2 krb5kdc[7097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:56 dist-test-slave-s0b2 krb5kdc[7097](info): set up 2 sockets
Sep 25 01:22:56 dist-test-slave-s0b2 krb5kdc[7097](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 01:22:57.051082 7090 mini_kdc.cc:121] Time spent starting KDC: real 1.592s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:22:57 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:22:57.207644 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37843
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:57.511458 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:57.511745 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:57.511875 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:57.518930 7110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:57.519030 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:57.519106 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:57.519186 7110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:57.519243 7110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:57.528841 7110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:57.530113 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:57.531421 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:57.542871 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:57.542912 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:57.542950 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:57.543561 7110 server_base.cc:1034] running on GCE node
I20260925 01:22:57.544312 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:57.545728 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:57.546953 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790299377546900 us; error 38 us; skew 500 ppm
I20260925 01:22:57.547266 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:57 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299377, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:57.550150 7110 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:22:57.551546 7110 webserver.cc:460] Webserver started at http://127.6.236.190:45881/ using document root <none> and password file <none>
I20260925 01:22:57.552189 7110 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:57.552285 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:57.552521 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:57.555397 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "0258d7c016c24d9fb83034ba101468ac"
format_stamp: "Formatted at 2026-09-25 01:22:57 on dist-test-slave-s0b2"
server_key: "f1540baa690959b63ae9eeadca29b23f"
server_key_iv: "b02d4404039292f194925a8f350a51b5"
server_key_version: "encryptionkey@0"
I20260925 01:22:57.556118 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "0258d7c016c24d9fb83034ba101468ac"
format_stamp: "Formatted at 2026-09-25 01:22:57 on dist-test-slave-s0b2"
server_key: "f1540baa690959b63ae9eeadca29b23f"
server_key_iv: "b02d4404039292f194925a8f350a51b5"
server_key_version: "encryptionkey@0"
I20260925 01:22:57.561234 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 01:22:57.564620 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:57.565934 7110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:22:57.566100 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "0258d7c016c24d9fb83034ba101468ac"
format_stamp: "Formatted at 2026-09-25 01:22:57 on dist-test-slave-s0b2"
server_key: "f1540baa690959b63ae9eeadca29b23f"
server_key_iv: "b02d4404039292f194925a8f350a51b5"
server_key_version: "encryptionkey@0"
I20260925 01:22:57.566321 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:57.584730 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:57.588681 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:57.588935 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:57.605744 7110 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:37843
I20260925 01:22:57.605767 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:37843 every 8 connection(s)
I20260925 01:22:57.607075 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:57.610764 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:57.612037 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7110
I20260925 01:22:57.612304 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:22:57.614070 7090 external_mini_cluster.cc:1428] Setting key db7e21804323739c10c3c487e0039815
I20260925 01:22:57.620837 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac: Bootstrap starting.
I20260925 01:22:57.623709 7177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:57.624632 7177 log.cc:824] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:57.627440 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac: No bootstrap required, opened a new log
I20260925 01:22:57.636116 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } }
I20260925 01:22:57.636543 7177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:57.636659 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0258d7c016c24d9fb83034ba101468ac, State: Initialized, Role: FOLLOWER
I20260925 01:22:57.637326 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } }
I20260925 01:22:57.637583 7177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:57.637715 7177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:57.637861 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:57.640081 7177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } }
I20260925 01:22:57.640550 7177 leader_election.cc:302] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0258d7c016c24d9fb83034ba101468ac; no voters:
I20260925 01:22:57.641115 7177 leader_election.cc:288] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:57.641222 7182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 01:22:57 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:22:57.642483 7182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [term 1 LEADER]: Becoming Leader. State: Replica: 0258d7c016c24d9fb83034ba101468ac, State: Running, Role: LEADER
I20260925 01:22:57.643092 7182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } }
I20260925 01:22:57.643473 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:57.646428 7184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0258d7c016c24d9fb83034ba101468ac. Latest consensus state: current_term: 1 leader_uuid: "0258d7c016c24d9fb83034ba101468ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } } }
I20260925 01:22:57.646884 7184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:57.646441 7183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0258d7c016c24d9fb83034ba101468ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0258d7c016c24d9fb83034ba101468ac" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37843 } } }
I20260925 01:22:57.647433 7183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:57.648008 7189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:57.654728 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:57.617718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.6.236.190:37843)
0925 01:22:57.618641 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:57.618656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:57.619869 (+ 1213us) server_negotiation.cc:407] Connection header received
0925 01:22:57.628870 (+ 9001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:57.628898 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:57.629359 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:57.629798 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:57.632436 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:57.634494 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:57.636039 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:57.637035 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:57.637209 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:57.643101 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:57.643112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:57.643127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:57.643167 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:57.647017 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:57.648103 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:57.648111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:57.648118 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:57.648330 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:57.648913 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:57.648917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:57.648920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:57.649678 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:57.652412 (+ 2734us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:57.652719 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 01:22:57.653178 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":398,"threads_started":1}
I20260925 01:22:57.659722 7189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:57.670403 7189 catalog_manager.cc:1347] Generated new cluster ID: d70d9932c60f4c15a74827f83720cbba
I20260925 01:22:57.670535 7189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:57.679956 7189 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:57.681474 7189 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:57.687692 7189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0258d7c016c24d9fb83034ba101468ac: Generated new TSK 0
I20260925 01:22:57.688428 7189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:22:57.727481 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:58.035727 7203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:58.036016 7203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:58.036154 7203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:58.043292 7203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:58.043416 7203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:58.043572 7203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:22:58.053347 7203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:58.054708 7203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:58.056020 7203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:58.068588 7208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.068563 7209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.068645 7211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:58.069499 7203 server_base.cc:1034] running on GCE node
I20260925 01:22:58.070076 7203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:58.071089 7203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:58.072284 7203 hybrid_clock.cc:648] HybridClock initialized: now 1790299378072224 us; error 49 us; skew 500 ppm
I20260925 01:22:58.072575 7203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:58 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:58.075527 7203 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:22:58.076825 7203 webserver.cc:460] Webserver started at http://127.6.236.129:41735/ using document root <none> and password file <none>
I20260925 01:22:58.077435 7203 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:58.077531 7203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:58.077783 7203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:58.080523 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "a94c13e0c01243b598e8ba3f588c2472"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "cb95c92aad905db155db17df5ff9a728"
server_key_iv: "600a9ad0bef66022b68db3d0770939d4"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.081197 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "a94c13e0c01243b598e8ba3f588c2472"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "cb95c92aad905db155db17df5ff9a728"
server_key_iv: "600a9ad0bef66022b68db3d0770939d4"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.086076 7203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:22:58.089371 7218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.090660 7203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:58.090794 7203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "a94c13e0c01243b598e8ba3f588c2472"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "cb95c92aad905db155db17df5ff9a728"
server_key_iv: "600a9ad0bef66022b68db3d0770939d4"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.090950 7203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:58.105161 7203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:58.109184 7203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:58.109411 7203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:58.110314 7203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:22:58.111560 7203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:22:58.111640 7203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.111740 7203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:22:58.111791 7203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.140185 7203 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:44869
I20260925 01:22:58.140220 7330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:44869 every 8 connection(s)
I20260925 01:22:58.141427 7203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:22:58.142956 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7203
I20260925 01:22:58.143172 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:22:58.143872 7090 external_mini_cluster.cc:1428] Setting key e1bfe30087ba779b7ff13df575d38d02
Sep 25 01:22:58 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:22:58.161878 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:58.143899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:39969 (local address 127.6.236.190:37843)
0925 01:22:58.144093 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:58.144099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:58.145376 (+ 1277us) server_negotiation.cc:407] Connection header received
0925 01:22:58.147269 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:58.147276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:58.147356 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:58.147458 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:58.148877 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.149803 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.151375 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.152007 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.152144 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:58.155638 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:58.155644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:58.155649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:58.155687 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:58.158905 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:58.159603 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:58.159609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:58.159613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:58.159775 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:58.160181 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:58.160186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:58.160189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:58.160585 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:58.160745 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:58.161451 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 01:22:58.161639 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:22:58.162851 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:58.144404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37843 (local address 127.6.236.129:39969)
0925 01:22:58.145220 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:58.145266 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:58.147004 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:58.147635 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:58.147643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:58.148135 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:58.148712 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:58.148731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.149941 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.149945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:58.151243 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:58.151256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.152215 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.152222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:58.152435 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:58.153103 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:58.153125 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:58.155393 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:58.159040 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:58.159047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:58.159061 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:58.159473 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:58.159870 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:58.159873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:58.159876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:58.160069 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:58.160695 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:58.160701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:58.161034 (+ 333us) client_negotiation.cc:769] Sending connection context
0925 01:22:58.161303 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 01:22:58.161724 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
I20260925 01:22:58.164562 7331 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37843
I20260925 01:22:58.164855 7331 heartbeater.cc:464] Registering TS with master...
I20260925 01:22:58.165603 7331 heartbeater.cc:511] Master 127.6.236.190:37843 requested a full tablet report, sending...
I20260925 01:22:58.167794 7142 ts_manager.cc:194] Registered new tserver with Master: a94c13e0c01243b598e8ba3f588c2472 (127.6.236.129:44869)
I20260925 01:22:58.169864 7142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:39969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:22:58.182794 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:58.488293 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:58.488564 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:58.488669 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:58.495812 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:58.495918 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:58.496043 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:22:58.505558 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:58.506882 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:58.508195 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:58.520630 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.520668 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.520689 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:58.521689 7336 server_base.cc:1034] running on GCE node
I20260925 01:22:58.522360 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:58.523255 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:58.524441 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790299378524390 us; error 43 us; skew 500 ppm
I20260925 01:22:58.524744 7336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:58 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:58.527648 7336 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:22:58.528968 7336 webserver.cc:460] Webserver started at http://127.6.236.130:46081/ using document root <none> and password file <none>
I20260925 01:22:58.529598 7336 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:58.529712 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:58.530000 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:58.532778 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "f1d7492613da4c20aff38b2476ce329b"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "b7a85c3ae87e0d9afe7771f492ea4267"
server_key_iv: "0489083a50bbc8d8053e573990699808"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.533489 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "f1d7492613da4c20aff38b2476ce329b"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "b7a85c3ae87e0d9afe7771f492ea4267"
server_key_iv: "0489083a50bbc8d8053e573990699808"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.538358 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:22:58.541491 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.542721 7336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 01:22:58.542856 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "f1d7492613da4c20aff38b2476ce329b"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "b7a85c3ae87e0d9afe7771f492ea4267"
server_key_iv: "0489083a50bbc8d8053e573990699808"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.543032 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:58.559068 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:58.562579 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:58.562813 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:58.563651 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:22:58.564985 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:22:58.565055 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.565145 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:22:58.565199 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.593519 7336 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:45635
I20260925 01:22:58.593565 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:45635 every 8 connection(s)
I20260925 01:22:58.594836 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:22:58.596306 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7336
I20260925 01:22:58.596505 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:22:58.596995 7090 external_mini_cluster.cc:1428] Setting key 9d827610c25427b0d45d5bdeb8c0684d
Sep 25 01:22:58 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:22:58.616449 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:58.597405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:52273 (local address 127.6.236.190:37843)
0925 01:22:58.597610 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:58.597616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:58.598894 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 01:22:58.600705 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:58.600711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:58.600789 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:58.600892 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:58.602308 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.603121 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.604796 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.605414 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.605544 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:58.609686 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:58.609693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:58.609697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:58.609742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:58.613162 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:58.613877 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:58.613883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:58.613886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:58.614066 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:58.614585 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:58.614590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:58.614594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:58.614948 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:58.615080 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:58.615865 (+ 785us) server_negotiation.cc:299] Negotiation successful
0925 01:22:58.616095 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:22:58.617571 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:58.597940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37843 (local address 127.6.236.130:52273)
0925 01:22:58.598743 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:58.598792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:58.600447 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:58.601076 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:58.601084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:58.601580 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:58.602132 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:58.602149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.603287 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.603292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:58.604645 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:58.604657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:58.605649 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:58.605657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:58.605943 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:58.606659 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:58.606693 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:58.609453 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:58.613299 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:58.613309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:58.613325 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:58.613772 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:58.614188 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:58.614192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:58.614195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:58.614477 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:58.615072 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:58.615080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:58.615496 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 01:22:58.615782 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 01:22:58.616299 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260925 01:22:58.618930 7464 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37843
I20260925 01:22:58.619204 7464 heartbeater.cc:464] Registering TS with master...
I20260925 01:22:58.619956 7464 heartbeater.cc:511] Master 127.6.236.190:37843 requested a full tablet report, sending...
I20260925 01:22:58.621505 7142 ts_manager.cc:194] Registered new tserver with Master: f1d7492613da4c20aff38b2476ce329b (127.6.236.130:45635)
I20260925 01:22:58.622617 7142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:52273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:22:58.634264 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:58.940586 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:58.940864 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:58.941001 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:58.948127 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:58.948246 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:58.948405 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:22:58.957919 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37843
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:58.959172 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:58.960465 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:58.972893 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.972849 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:58.972918 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:58.973843 7469 server_base.cc:1034] running on GCE node
I20260925 01:22:58.974436 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:58.975327 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:58.976544 7469 hybrid_clock.cc:648] HybridClock initialized: now 1790299378976491 us; error 47 us; skew 500 ppm
I20260925 01:22:58.976828 7469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:58 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:58.979710 7469 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:22:58.981006 7469 webserver.cc:460] Webserver started at http://127.6.236.131:44717/ using document root <none> and password file <none>
I20260925 01:22:58.981652 7469 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:58.981747 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:58.981988 7469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:58.984763 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "24547fb9a3c14345ad75a750d5e052d8"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "c71af0196ee0cda071c5ca073b4c8635"
server_key_iv: "0f467ee672fd4b9b998435c44eb50623"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.985633 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "24547fb9a3c14345ad75a750d5e052d8"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "c71af0196ee0cda071c5ca073b4c8635"
server_key_iv: "0f467ee672fd4b9b998435c44eb50623"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.990670 7469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:22:58.993805 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:58.995151 7469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:58.995275 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "24547fb9a3c14345ad75a750d5e052d8"
format_stamp: "Formatted at 2026-09-25 01:22:58 on dist-test-slave-s0b2"
server_key: "c71af0196ee0cda071c5ca073b4c8635"
server_key_iv: "0f467ee672fd4b9b998435c44eb50623"
server_key_version: "encryptionkey@0"
I20260925 01:22:58.995432 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:59.024060 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:59.027942 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:59.028187 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:59.029263 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:22:59.030596 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:22:59.030668 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:59.030754 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:22:59.030812 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:59.060283 7469 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:35631
I20260925 01:22:59.060328 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:35631 every 8 connection(s)
I20260925 01:22:59.061569 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:22:59.067109 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7469
I20260925 01:22:59.067286 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:22:59.067729 7090 external_mini_cluster.cc:1428] Setting key ed30da3344cae78a5befe02d1166ac1f
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:22:59.080324 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.064020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57971 (local address 127.6.236.190:37843)
0925 01:22:59.064189 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.064196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.065363 (+ 1167us) server_negotiation.cc:407] Connection header received
0925 01:22:59.067210 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.067216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.067297 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.067382 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.068959 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.069805 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.071458 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.071857 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.071972 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.075139 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.075146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.075149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.075181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.077497 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.078154 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.078159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.078163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.078325 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.078724 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.078729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.078733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.079010 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.079140 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.079778 (+ 638us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.080013 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:22:59.081256 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.064507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37843 (local address 127.6.236.131:57971)
0925 01:22:59.065219 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:59.065256 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:59.066962 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:59.067630 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:59.067639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:59.068158 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:59.068805 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:59.068825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.069951 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.069955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:59.071331 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:59.071345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.072057 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.072064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:59.072282 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.072876 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:59.072896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:59.074939 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:59.077631 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:59.077639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:59.077655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:59.078038 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:59.078421 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:59.078426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:59.078429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:59.078614 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:59.079128 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:59.079135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:59.079440 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 01:22:59.079666 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 01:22:59.080164 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":301,"threads_started":1}
I20260925 01:22:59.082677 7598 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37843
I20260925 01:22:59.082983 7598 heartbeater.cc:464] Registering TS with master...
I20260925 01:22:59.083715 7598 heartbeater.cc:511] Master 127.6.236.190:37843 requested a full tablet report, sending...
I20260925 01:22:59.085000 7141 ts_manager.cc:194] Registered new tserver with Master: 24547fb9a3c14345ad75a750d5e052d8 (127.6.236.131:35631)
I20260925 01:22:59.085803 7141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:57971
I20260925 01:22:59.095593 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:22:59.136674 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.122298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.6.236.129:44869)
0925 01:22:59.122855 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.122865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.122972 (+ 107us) server_negotiation.cc:407] Connection header received
0925 01:22:59.123219 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.123229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.123457 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.123609 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.124965 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.125919 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.127180 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.127720 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.127838 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.131402 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.131412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.131425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.131456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.133966 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.134640 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.134647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.134649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.134775 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.135158 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.135162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.135164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.135501 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.135799 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.136079 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.136332 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":271,"threads_started":1}
W20260925 01:22:59.137851 7246 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53500
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 01:22:59.172879 7331 heartbeater.cc:503] Master 127.6.236.190:37843 was elected leader, sending a full tablet report...
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:22:59.179370 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.166628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.6.236.129:44869)
0925 01:22:59.166849 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.166856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.166994 (+ 138us) server_negotiation.cc:407] Connection header received
0925 01:22:59.167164 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.167170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.167273 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.167387 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.168570 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.169267 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.170550 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.170990 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.171126 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.174953 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.174958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.174961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.174991 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.177267 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.177799 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.177802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.177805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.177910 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.178166 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.178169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.178171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.178440 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.178606 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.178915 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.179083 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:22:59.180749 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7203
I20260925 01:22:59.193694 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.181610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.6.236.129:44869)
0925 01:22:59.181751 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.181755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.182152 (+ 397us) server_negotiation.cc:407] Connection header received
0925 01:22:59.182260 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.182266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.182334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.182422 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.183497 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.184174 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.185326 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.185731 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.185838 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.187889 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.187893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.187896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.187924 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.190770 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.191520 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.191524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.191526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.191642 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.191995 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.191999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.192001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.192316 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.192574 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.193047 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.193245 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:22:59.194702 7326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:59.276724 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7203
I20260925 01:22:59.291011 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7336
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:22:59.306449 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.291953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.6.236.130:45635)
0925 01:22:59.292449 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.292456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.292480 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:22:59.292544 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.292550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.292755 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.292905 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.294091 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.295146 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.296355 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.296893 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.297008 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.300810 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.300821 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.300835 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.300870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.303757 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.304371 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.304378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.304380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.304505 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.304880 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.304884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.304886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.305233 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.305524 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.305819 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.306087 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":260,"threads_started":1}
I20260925 01:22:59.307703 7459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:59.391111 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7336
I20260925 01:22:59.410009 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7469
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:22:59.425414 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:59.411048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.6.236.131:35631)
0925 01:22:59.411733 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:59.411740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:59.411768 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:22:59.411830 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:59.411837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:59.412055 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:59.412195 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:59.413366 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.414538 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.415772 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:59.416340 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:59.416439 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:59.420441 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:59.420453 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:59.420478 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:59.420513 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:59.423112 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.423750 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.423757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.423759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.423882 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:59.424160 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:59.424163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:59.424165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:59.424505 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:59.424727 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:59.424940 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 01:22:59.425134 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":366,"threads_started":1}
I20260925 01:22:59.426743 7593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:59.507428 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7469
I20260925 01:22:59.521400 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7110
I20260925 01:22:59.522603 7172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:59.623263 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7110
2026-09-25T01:22:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4213 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7619](info): set up 2 sockets
Sep 25 01:22:59 dist-test-slave-s0b2 krb5kdc[7619](info): commencing operation
krb5kdc: starting...
W20260925 01:23:00.602970 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.004s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:00 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:00.759641 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:41635
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:01.062742 7632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:01.063027 7632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:01.063158 7632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:01.070144 7632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:01.070277 7632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:01.070358 7632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:01.070430 7632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:01.070494 7632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:01.080255 7632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:01.081558 7632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:01.082928 7632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:01.094278 7637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:01.094202 7638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:01.094293 7640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:01.094969 7632 server_base.cc:1034] running on GCE node
I20260925 01:23:01.095636 7632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:01.096941 7632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:01.098153 7632 hybrid_clock.cc:648] HybridClock initialized: now 1790299381098110 us; error 36 us; skew 500 ppm
I20260925 01:23:01.098516 7632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:01 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:01.101514 7632 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:01.102942 7632 webserver.cc:460] Webserver started at http://127.6.236.190:39617/ using document root <none> and password file <none>
I20260925 01:23:01.103617 7632 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:01.103724 7632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:01.104022 7632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:01.106966 7632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "e82f1b592aa84a3a91d3486aa8d71452"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "a553af1ebefb2a45083c238757482235"
server_key_iv: "172994b4ceecbb27fcecc8bfaa514141"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.107849 7632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "e82f1b592aa84a3a91d3486aa8d71452"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "a553af1ebefb2a45083c238757482235"
server_key_iv: "172994b4ceecbb27fcecc8bfaa514141"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.112926 7632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:01.116299 7647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:01.117544 7632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:01.117689 7632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "e82f1b592aa84a3a91d3486aa8d71452"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "a553af1ebefb2a45083c238757482235"
server_key_iv: "172994b4ceecbb27fcecc8bfaa514141"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.117864 7632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:01.145509 7632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:01.149413 7632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:01.149685 7632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:01.166404 7632 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:41635
I20260925 01:23:01.166406 7698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:41635 every 8 connection(s)
I20260925 01:23:01.167771 7632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:01.171485 7699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:01.172998 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7632
I20260925 01:23:01.173192 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:01.173789 7090 external_mini_cluster.cc:1428] Setting key 8f79853494d1006f221609ad7d62081f
I20260925 01:23:01.182171 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452: Bootstrap starting.
I20260925 01:23:01.185168 7699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:01.186127 7699 log.cc:824] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:23:01 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:01.189378 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452: No bootstrap required, opened a new log
I20260925 01:23:01.196384 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:01.175784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.6.236.190:41635)
0925 01:23:01.176512 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:01.176529 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:01.176571 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:23:01.178208 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:01.178230 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:01.178644 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:01.178985 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:01.180340 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.181622 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.182868 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.183532 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.183701 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:01.187703 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:01.187714 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:01.187730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:01.187772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:01.191675 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:01.192537 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:01.192546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:01.192553 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:01.192773 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:01.193118 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:01.193124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:01.193127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:01.193662 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:01.193952 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:01.194445 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 01:23:01.195032 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":298,"threads_started":1}
I20260925 01:23:01.199420 7699 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } }
I20260925 01:23:01.199796 7699 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:01.199889 7699 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82f1b592aa84a3a91d3486aa8d71452, State: Initialized, Role: FOLLOWER
I20260925 01:23:01.200402 7699 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } }
I20260925 01:23:01.200613 7699 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:01.200731 7699 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:01.200867 7699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:01.203215 7699 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } }
I20260925 01:23:01.203684 7699 leader_election.cc:302] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82f1b592aa84a3a91d3486aa8d71452; no voters:
I20260925 01:23:01.204201 7699 leader_election.cc:288] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:01.204325 7704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:01.205335 7704 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [term 1 LEADER]: Becoming Leader. State: Replica: e82f1b592aa84a3a91d3486aa8d71452, State: Running, Role: LEADER
I20260925 01:23:01.205847 7704 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } }
I20260925 01:23:01.206214 7699 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:01.208663 7706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82f1b592aa84a3a91d3486aa8d71452. Latest consensus state: current_term: 1 leader_uuid: "e82f1b592aa84a3a91d3486aa8d71452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } } }
I20260925 01:23:01.209426 7706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:01.208989 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82f1b592aa84a3a91d3486aa8d71452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82f1b592aa84a3a91d3486aa8d71452" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41635 } } }
I20260925 01:23:01.209695 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:01.215379 7713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:01.219380 7713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:01.227654 7713 catalog_manager.cc:1347] Generated new cluster ID: ad0995888b194f6ca635f9b32bd688b7
I20260925 01:23:01.227761 7713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:01.238508 7713 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:01.239787 7713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:01.251720 7713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82f1b592aa84a3a91d3486aa8d71452: Generated new TSK 0
I20260925 01:23:01.252554 7713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:01.297923 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:01.638229 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:01.638547 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:01.638695 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:01.645779 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:01.645900 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:01.646051 7725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:01.655699 7725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:01.657096 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:01.658507 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:01.671481 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:01.671427 7733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:01.671387 7731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:01.672025 7725 server_base.cc:1034] running on GCE node
I20260925 01:23:01.672647 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:01.673511 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:01.674710 7725 hybrid_clock.cc:648] HybridClock initialized: now 1790299381674666 us; error 33 us; skew 500 ppm
I20260925 01:23:01.675005 7725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:01 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:01.677955 7725 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:01.679337 7725 webserver.cc:460] Webserver started at http://127.6.236.129:38479/ using document root <none> and password file <none>
I20260925 01:23:01.680009 7725 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:01.680123 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:01.680410 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:01.683267 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "adfa067f142a4fa5b4b5badce4df4157"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "6af666e69834a4336d7dad7aefb8ab4b"
server_key_iv: "6ace017eb95a04aee241399bfd4def97"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.684007 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "adfa067f142a4fa5b4b5badce4df4157"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "6af666e69834a4336d7dad7aefb8ab4b"
server_key_iv: "6ace017eb95a04aee241399bfd4def97"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.688957 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:01.692528 7740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:01.693868 7725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:01.694012 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "adfa067f142a4fa5b4b5badce4df4157"
format_stamp: "Formatted at 2026-09-25 01:23:01 on dist-test-slave-s0b2"
server_key: "6af666e69834a4336d7dad7aefb8ab4b"
server_key_iv: "6ace017eb95a04aee241399bfd4def97"
server_key_version: "encryptionkey@0"
I20260925 01:23:01.694192 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:01.725766 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:01.729643 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:01.729885 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:01.730865 7725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:01.732097 7725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:01.732167 7725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:01.732286 7725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:01.732339 7725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:01.760226 7725 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:35209
I20260925 01:23:01.760282 7852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:35209 every 8 connection(s)
I20260925 01:23:01.761765 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:01.765761 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7725
I20260925 01:23:01.765983 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:01.766790 7090 external_mini_cluster.cc:1428] Setting key 40dc4cccb21e8e1947578750c5928161
Sep 25 01:23:01 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:01.783023 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:01.764343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:32779 (local address 127.6.236.190:41635)
0925 01:23:01.764744 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:01.764748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:01.765826 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 01:23:01.768253 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:01.768259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:01.768356 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:01.768448 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:01.770436 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.771130 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.773140 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.773531 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.773615 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:01.777578 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:01.777583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:01.777586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:01.777618 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:01.779928 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:01.780787 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:01.780791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:01.780793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:01.780905 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:01.781361 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:01.781364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:01.781366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:01.781619 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:01.781720 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:01.782600 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 01:23:01.782762 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":202,"threads_started":1}
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:23:01.784425 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:01.764857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41635 (local address 127.6.236.129:32779)
0925 01:23:01.765623 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:01.765688 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:01.767985 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:01.768720 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:01.768730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:01.769463 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:01.770265 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:01.770293 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.771281 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.771286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:01.772995 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:01.773009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:01.773757 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:01.773764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:01.774041 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:01.774730 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:01.774765 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:01.777365 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:01.780062 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:01.780078 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:01.780098 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:01.780662 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:01.781007 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:01.781011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:01.781015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:01.781257 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:01.781737 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:01.781745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:01.782179 (+ 434us) client_negotiation.cc:769] Sending connection context
0925 01:23:01.782506 (+ 327us) client_negotiation.cc:240] Negotiation successful
0925 01:23:01.783043 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260925 01:23:01.785867 7853 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
I20260925 01:23:01.786182 7853 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:01.786908 7853 heartbeater.cc:511] Master 127.6.236.190:41635 requested a full tablet report, sending...
I20260925 01:23:01.788868 7664 ts_manager.cc:194] Registered new tserver with Master: adfa067f142a4fa5b4b5badce4df4157 (127.6.236.129:35209)
I20260925 01:23:01.790650 7664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:32779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:01.804992 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:02.106909 7859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:02.107167 7859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:02.107291 7859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:02.114319 7859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:02.114426 7859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:02.114575 7859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:02.123940 7859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:02.125174 7859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:02.126477 7859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:02.139211 7864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:02.139251 7867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:02.139285 7865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:02.140287 7859 server_base.cc:1034] running on GCE node
I20260925 01:23:02.140872 7859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:02.141819 7859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:02.143015 7859 hybrid_clock.cc:648] HybridClock initialized: now 1790299382142966 us; error 37 us; skew 500 ppm
I20260925 01:23:02.143312 7859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:02 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:02.146296 7859 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:02.147768 7859 webserver.cc:460] Webserver started at http://127.6.236.130:46867/ using document root <none> and password file <none>
I20260925 01:23:02.148430 7859 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:02.148545 7859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:02.148839 7859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:02.151701 7859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "44ebcf6005df4a9b8b656448a7aa1688"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "fbcf7cded3c11f34f3b62c0456ba0c39"
server_key_iv: "a5c1cdd1845b62a889547bee39a9cc74"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.152442 7859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "44ebcf6005df4a9b8b656448a7aa1688"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "fbcf7cded3c11f34f3b62c0456ba0c39"
server_key_iv: "a5c1cdd1845b62a889547bee39a9cc74"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.157356 7859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:23:02.160617 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.161901 7859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:02.162046 7859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "44ebcf6005df4a9b8b656448a7aa1688"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "fbcf7cded3c11f34f3b62c0456ba0c39"
server_key_iv: "a5c1cdd1845b62a889547bee39a9cc74"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.162225 7859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:02.178922 7859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:02.182731 7859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:02.182984 7859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:02.183866 7859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:02.185146 7859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:02.185222 7859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.185344 7859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:02.185397 7859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.214924 7859 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36069
I20260925 01:23:02.214982 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36069 every 8 connection(s)
I20260925 01:23:02.216275 7859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:02.219280 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7859
I20260925 01:23:02.219477 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:02.220019 7090 external_mini_cluster.cc:1428] Setting key d1e556f4f9eb351ed99c062e7c902613
Sep 25 01:23:02 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:02.235961 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.218637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:59627 (local address 127.6.236.190:41635)
0925 01:23:02.218796 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.218801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.220255 (+ 1454us) server_negotiation.cc:407] Connection header received
0925 01:23:02.222319 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.222324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.222406 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.222491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:02.223908 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.224613 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.226076 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.226523 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.226630 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.230148 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:02.230153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:02.230156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:02.230188 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:02.233079 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.233747 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.233752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.233756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.233908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.234297 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.234301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.234305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.234662 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:02.234804 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.235420 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.235643 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:23:02.236894 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.219195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41635 (local address 127.6.236.130:59627)
0925 01:23:02.219956 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:02.219996 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:02.222057 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:02.222733 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:02.222741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:02.223220 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:02.223754 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.223772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.224752 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.224756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.225944 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.225956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.226732 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.226740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.226941 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.227592 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:02.227625 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:02.229936 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:02.233211 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:02.233218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:02.233231 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:02.233637 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:02.234007 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:02.234010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:02.234013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:02.234182 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:02.234772 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:02.234777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:02.235088 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 01:23:02.235312 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 01:23:02.235736 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":323,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:23:02.238337 7987 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41635
I20260925 01:23:02.238672 7987 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:02.239496 7987 heartbeater.cc:511] Master 127.6.236.190:41635 requested a full tablet report, sending...
I20260925 01:23:02.241008 7664 ts_manager.cc:194] Registered new tserver with Master: 44ebcf6005df4a9b8b656448a7aa1688 (127.6.236.130:36069)
I20260925 01:23:02.241787 7664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:59627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:02.255276 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:02.558187 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:02.558471 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:02.558580 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:02.565572 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:02.565672 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:02.565793 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:02.575212 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41635
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:02.576406 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:02.577693 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:02.589896 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:02.589855 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:02.589929 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:02.590634 7992 server_base.cc:1034] running on GCE node
I20260925 01:23:02.591192 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:02.592059 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:02.593271 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790299382593214 us; error 51 us; skew 500 ppm
I20260925 01:23:02.593566 7992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:02 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:02.596540 7992 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:02.597903 7992 webserver.cc:460] Webserver started at http://127.6.236.131:42691/ using document root <none> and password file <none>
I20260925 01:23:02.598616 7992 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:02.598714 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:02.598953 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:02.601652 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "5aabcc88ae8c496889be4627d1bb1ce3"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "e27811c293ff42481fd560b91cb49b3e"
server_key_iv: "cdc932dd75530998add92d452ff8ed5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.602422 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "5aabcc88ae8c496889be4627d1bb1ce3"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "e27811c293ff42481fd560b91cb49b3e"
server_key_iv: "cdc932dd75530998add92d452ff8ed5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.607193 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:23:02.610663 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.611958 7992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:23:02.612087 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "5aabcc88ae8c496889be4627d1bb1ce3"
format_stamp: "Formatted at 2026-09-25 01:23:02 on dist-test-slave-s0b2"
server_key: "e27811c293ff42481fd560b91cb49b3e"
server_key_iv: "cdc932dd75530998add92d452ff8ed5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:02.612241 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:02.651934 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:02.655884 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:02.656121 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:02.657019 7992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:02.658385 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:02.658454 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.658533 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:02.658582 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:02.687783 7992 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:33259
I20260925 01:23:02.687829 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:33259 every 8 connection(s)
I20260925 01:23:02.689020 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:02.698594 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 7992
I20260925 01:23:02.698798 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:02.699339 7090 external_mini_cluster.cc:1428] Setting key c8523be8b9d5686235ff4a93369eb114
Sep 25 01:23:02 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299382, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:02.709182 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.691434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:49337 (local address 127.6.236.190:41635)
0925 01:23:02.691630 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.691635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.692834 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 01:23:02.694751 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.694758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.694846 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.694936 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:02.696598 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.697392 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.699534 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.700249 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.700379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.704128 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:02.704133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:02.704136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:02.704171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:02.706522 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.707195 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.707200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.707204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.707324 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.707719 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.707722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.707724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.708003 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:02.708115 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.708756 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.708943 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:02.710157 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.691970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41635 (local address 127.6.236.131:49337)
0925 01:23:02.692666 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:02.692711 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:02.694452 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:02.695199 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:02.695209 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:02.695810 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:02.696414 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.696433 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.697555 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.697559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.699247 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.699263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.700525 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.700534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.700775 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.701586 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:02.701613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:02.703917 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:02.706641 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:02.706650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:02.706669 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:02.707074 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:02.707425 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:02.707429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:02.707431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:02.707630 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:02.708100 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:02.708106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:02.708420 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 01:23:02.708652 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 01:23:02.709029 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":255,"threads_started":1}
I20260925 01:23:02.711467 8120 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41635
I20260925 01:23:02.711756 8120 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:02.712435 8120 heartbeater.cc:511] Master 127.6.236.190:41635 requested a full tablet report, sending...
I20260925 01:23:02.713775 7664 ts_manager.cc:194] Registered new tserver with Master: 5aabcc88ae8c496889be4627d1bb1ce3 (127.6.236.131:33259)
I20260925 01:23:02.714694 7664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:49337
I20260925 01:23:02.715524 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:23:02.732570 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.721417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.6.236.190:41635)
0925 01:23:02.721626 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.721632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.722044 (+ 412us) server_negotiation.cc:407] Connection header received
0925 01:23:02.722209 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.722213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.722323 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.722396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:02.723795 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.724545 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.725701 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.726062 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.726146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.728249 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:02.728253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:02.728257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:02.728284 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:02.730510 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.731044 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.731047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.731049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.731163 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:02.731417 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:02.731420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:02.731422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:02.731707 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:02.731872 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.732122 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.732264 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 01:23:02.738791 7663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:02.745342 7663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:02.784397 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.756820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:33259 (local address 127.0.0.1:48264)
0925 01:23:02.765475 (+ 8655us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:02.765517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:02.765692 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:02.777755 (+ 12063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:02.777759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:02.777784 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:02.778151 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.778164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.779953 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.779958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.781871 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.781883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.783356 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.783362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.783527 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.783581 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 01:23:02.783716 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:23:02.783899 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8403,"mutex_wait_us":71,"thread_start_us":18714,"threads_started":1}
I20260925 01:23:02.785045 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.758985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35209 (local address 127.0.0.1:35744)
0925 01:23:02.777464 (+ 18479us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:02.777481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:02.777611 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:02.778324 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:02.778329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:02.778352 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:02.778660 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.778670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.780304 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.780310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.782433 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.782447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.784339 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.784346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.784519 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.784550 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:23:02.784624 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 01:23:02.784737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18333,"thread_start_us":6109,"threads_started":1}
I20260925 01:23:02.785322 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.756913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.6.236.131:33259)
0925 01:23:02.777112 (+ 20199us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.777119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.777148 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:23:02.777328 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.777341 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.777574 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.778376 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:02.778633 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.779830 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.782015 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.783158 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.783282 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.783492 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.784792 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.784953 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20028,"thread_start_us":19879,"threads_started":1}
I20260925 01:23:02.785365 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.756864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36069 (local address 127.0.0.1:54494)
0925 01:23:02.775797 (+ 18933us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:02.775815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:02.775931 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:02.776991 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:02.776999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:02.777031 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:02.777524 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.777540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.779362 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.779367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.781880 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:02.781894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.783619 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.783625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:02.783785 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.783813 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 01:23:02.783884 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 01:23:02.783979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18809,"mutex_wait_us":23,"thread_start_us":20306,"threads_started":1}
I20260925 01:23:02.785682 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.757105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.6.236.130:36069)
0925 01:23:02.776097 (+ 18992us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.776105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.776132 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:23:02.776228 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.776236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.776504 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.776703 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:02.777726 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.779218 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.782072 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.783157 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.783280 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.783463 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.785235 (+ 1772us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.785404 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18826,"thread_start_us":18904,"threads_started":1}
I20260925 01:23:02.785749 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:02.759497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.6.236.129:35209)
0925 01:23:02.776636 (+ 17139us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:02.776644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:02.777736 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 01:23:02.777827 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:02.777834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:02.778081 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:02.778292 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:02.778814 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.780108 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.782588 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:02.783840 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:02.783980 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:02.784183 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:02.785298 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0925 01:23:02.785458 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16978,"thread_start_us":16850,"threads_started":1}
I20260925 01:23:02.788923 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 64ba8b694e384608aadc30b05fee15b6 (DEFAULT_TABLE table=test-table [id=27087a5f204f426faa1c7c2673c012cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:02.789070 7922 tablet_service.cc:1467] Processing CreateTablet for tablet 64ba8b694e384608aadc30b05fee15b6 (DEFAULT_TABLE table=test-table [id=27087a5f204f426faa1c7c2673c012cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:02.789070 7788 tablet_service.cc:1467] Processing CreateTablet for tablet 64ba8b694e384608aadc30b05fee15b6 (DEFAULT_TABLE table=test-table [id=27087a5f204f426faa1c7c2673c012cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:02.790132 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ba8b694e384608aadc30b05fee15b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:02.790364 7788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ba8b694e384608aadc30b05fee15b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:02.790375 7922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ba8b694e384608aadc30b05fee15b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:02.793885 7853 heartbeater.cc:503] Master 127.6.236.190:41635 was elected leader, sending a full tablet report...
I20260925 01:23:02.802021 8138 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Bootstrap starting.
I20260925 01:23:02.804241 8139 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Bootstrap starting.
I20260925 01:23:02.805593 8138 tablet_bootstrap.cc:654] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:02.806495 8140 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Bootstrap starting.
I20260925 01:23:02.807004 8138 log.cc:824] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:02.809441 8139 tablet_bootstrap.cc:654] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:02.809617 8138 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: No bootstrap required, opened a new log
I20260925 01:23:02.809814 8140 tablet_bootstrap.cc:654] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:02.809862 8138 ts_tablet_manager.cc:1397] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 01:23:02.810722 8139 log.cc:824] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:02.810956 8140 log.cc:824] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:02.813246 8139 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: No bootstrap required, opened a new log
I20260925 01:23:02.813491 8139 ts_tablet_manager.cc:1397] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:23:02.813602 8140 tablet_bootstrap.cc:492] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: No bootstrap required, opened a new log
I20260925 01:23:02.813843 8140 ts_tablet_manager.cc:1397] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 01:23:02.820047 8138 raft_consensus.cc:348] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.820472 8138 raft_consensus.cc:374] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:02.820562 8138 raft_consensus.cc:729] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aabcc88ae8c496889be4627d1bb1ce3, State: Initialized, Role: FOLLOWER
I20260925 01:23:02.820969 8139 raft_consensus.cc:348] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.821127 8138 consensus_queue.cc:260] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.821383 8139 raft_consensus.cc:374] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:02.821506 8139 raft_consensus.cc:729] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ebcf6005df4a9b8b656448a7aa1688, State: Initialized, Role: FOLLOWER
I20260925 01:23:02.822288 8120 heartbeater.cc:503] Master 127.6.236.190:41635 was elected leader, sending a full tablet report...
I20260925 01:23:02.822193 8139 consensus_queue.cc:260] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.822894 8138 ts_tablet_manager.cc:1428] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 01:23:02.823390 7987 heartbeater.cc:503] Master 127.6.236.190:41635 was elected leader, sending a full tablet report...
I20260925 01:23:02.823894 8139 ts_tablet_manager.cc:1428] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 01:23:02.825038 8140 raft_consensus.cc:348] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.825798 8140 raft_consensus.cc:374] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:02.825904 8140 raft_consensus.cc:729] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfa067f142a4fa5b4b5badce4df4157, State: Initialized, Role: FOLLOWER
I20260925 01:23:02.826508 8140 consensus_queue.cc:260] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:02.827800 8140 ts_tablet_manager.cc:1428] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260925 01:23:02.940477 8121 tablet.cc:2369] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:23:02.968039 7988 tablet.cc:2369] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:23:03.014209 7854 tablet.cc:2369] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:03.032001 8144 raft_consensus.cc:482] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:03.032330 8144 raft_consensus.cc:504] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:03.034011 8144 leader_election.cc:288] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ebcf6005df4a9b8b656448a7aa1688 (127.6.236.130:36069), adfa067f142a4fa5b4b5badce4df4157 (127.6.236.129:35209)
I20260925 01:23:03.039563 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.034549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36697 (local address 127.6.236.129:35209)
0925 01:23:03.034792 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.034798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.034983 (+ 185us) server_negotiation.cc:407] Connection header received
0925 01:23:03.035159 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.035164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.035226 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.035297 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:03.035724 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.036431 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.037973 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.038762 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.038912 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.038995 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.039220 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.039338 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 01:23:03.039654 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.034552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35209 (local address 127.6.236.131:36697)
0925 01:23:03.034883 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:03.034900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:03.035027 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:03.035344 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:03.035349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:03.035374 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:03.035595 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:03.035602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.036557 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.036562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:03.037827 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:03.037838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.038903 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.038909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:03.039082 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.039106 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:23:03.039185 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 01:23:03.039308 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"spinlock_wait_cycles":42752}
I20260925 01:23:03.040141 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.034564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:39201 (local address 127.6.236.130:36069)
0925 01:23:03.034793 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.034800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.035423 (+ 623us) server_negotiation.cc:407] Connection header received
0925 01:23:03.035608 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.035612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.035670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.035745 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:03.036256 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.036923 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.038532 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.039291 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.039436 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.039504 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.039820 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.039944 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:23:03.040642 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.034594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36069 (local address 127.6.236.131:39201)
0925 01:23:03.035317 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:03.035331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:03.035471 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:03.035766 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:03.035771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:03.035792 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:03.036099 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:03.036108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.037061 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.037066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:03.038382 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:03.038394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.039479 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.039485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:03.039637 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.039670 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:23:03.040301 (+ 631us) client_negotiation.cc:240] Negotiation successful
0925 01:23:03.040415 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":388,"threads_started":1}
I20260925 01:23:03.040604 7808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ba8b694e384608aadc30b05fee15b6" candidate_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfa067f142a4fa5b4b5badce4df4157" is_pre_election: true
I20260925 01:23:03.041098 7808 raft_consensus.cc:2454] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aabcc88ae8c496889be4627d1bb1ce3 in term 0.
I20260925 01:23:03.041606 7942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ba8b694e384608aadc30b05fee15b6" candidate_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ebcf6005df4a9b8b656448a7aa1688" is_pre_election: true
I20260925 01:23:03.041852 8008 leader_election.cc:302] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aabcc88ae8c496889be4627d1bb1ce3, adfa067f142a4fa5b4b5badce4df4157; no voters:
I20260925 01:23:03.042162 7942 raft_consensus.cc:2454] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aabcc88ae8c496889be4627d1bb1ce3 in term 0.
I20260925 01:23:03.042229 8144 raft_consensus.cc:2781] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:03.042387 8144 raft_consensus.cc:482] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:03.042475 8144 raft_consensus.cc:3037] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:03.044621 8144 raft_consensus.cc:504] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:03.045122 8144 leader_election.cc:288] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 44ebcf6005df4a9b8b656448a7aa1688 (127.6.236.130:36069), adfa067f142a4fa5b4b5badce4df4157 (127.6.236.129:35209)
I20260925 01:23:03.045533 7942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ba8b694e384608aadc30b05fee15b6" candidate_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ebcf6005df4a9b8b656448a7aa1688"
I20260925 01:23:03.045534 7808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ba8b694e384608aadc30b05fee15b6" candidate_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfa067f142a4fa5b4b5badce4df4157"
I20260925 01:23:03.045699 7942 raft_consensus.cc:3037] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:03.045696 7808 raft_consensus.cc:3037] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:03.047814 7808 raft_consensus.cc:2454] T 64ba8b694e384608aadc30b05fee15b6 P adfa067f142a4fa5b4b5badce4df4157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aabcc88ae8c496889be4627d1bb1ce3 in term 1.
I20260925 01:23:03.047837 7942 raft_consensus.cc:2454] T 64ba8b694e384608aadc30b05fee15b6 P 44ebcf6005df4a9b8b656448a7aa1688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aabcc88ae8c496889be4627d1bb1ce3 in term 1.
I20260925 01:23:03.048167 8008 leader_election.cc:302] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aabcc88ae8c496889be4627d1bb1ce3, adfa067f142a4fa5b4b5badce4df4157; no voters:
I20260925 01:23:03.048419 8144 raft_consensus.cc:2781] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:03.048797 8144 raft_consensus.cc:686] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 5aabcc88ae8c496889be4627d1bb1ce3, State: Running, Role: LEADER
I20260925 01:23:03.049222 8144 consensus_queue.cc:237] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } }
I20260925 01:23:03.053612 7663 catalog_manager.cc:5495] T 64ba8b694e384608aadc30b05fee15b6 P 5aabcc88ae8c496889be4627d1bb1ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aabcc88ae8c496889be4627d1bb1ce3 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aabcc88ae8c496889be4627d1bb1ce3" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44ebcf6005df4a9b8b656448a7aa1688" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adfa067f142a4fa5b4b5badce4df4157" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35209 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:03.082549 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.069388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.6.236.129:35209)
0925 01:23:03.069570 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.069588 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.070068 (+ 480us) server_negotiation.cc:407] Connection header received
0925 01:23:03.070198 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.070201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.070294 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.070369 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.071733 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.072447 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.073597 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.073987 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.074070 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.077614 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.077628 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.077643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.077678 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.080336 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.080887 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.080893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.080896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.081022 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.081290 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.081292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.081294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.081640 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.081854 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.082035 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.082185 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:03.123529 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.110476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.6.236.129:35209)
0925 01:23:03.110639 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.110646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.111045 (+ 399us) server_negotiation.cc:407] Connection header received
0925 01:23:03.111231 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.111237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.111308 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.111390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.112644 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.113351 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.114628 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.115055 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.115166 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.118787 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.118791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.118794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.118826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.121122 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.121672 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.121678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.121682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.121808 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.122194 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.122199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.122202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.122500 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.122739 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.123000 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.123208 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:03.163090 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.150048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.6.236.129:35209)
0925 01:23:03.150281 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.150288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.150556 (+ 268us) server_negotiation.cc:407] Connection header received
0925 01:23:03.150731 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.150735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.150800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.150886 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.152186 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.152869 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.154054 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.154514 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.154636 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.158184 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.158189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.158192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.158232 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.160666 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.161247 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.161252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.161254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.161365 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.161734 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.161738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.161741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.162021 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.162287 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.162560 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.162788 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 01:23:03.164721 7768 tablet_service.cc:2908] Rejecting scan request for tablet 64ba8b694e384608aadc30b05fee15b6: Uninitialized: clean time has not yet been initialized
I20260925 01:23:03.167572 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7725
I20260925 01:23:03.180907 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.169089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.6.236.129:35209)
0925 01:23:03.169238 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.169243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.170619 (+ 1376us) server_negotiation.cc:407] Connection header received
0925 01:23:03.170761 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.170764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.170829 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.170904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.171976 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.172624 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.173778 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.174164 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.174304 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.176251 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.176256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.176259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.176287 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.178591 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.179193 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.179197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.179199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.179312 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.179638 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.179643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.179645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.179886 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.180168 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.180430 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.180639 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:23:03.181561 7848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:03.270454 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7725
I20260925 01:23:03.284586 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7859
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:03.298429 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.285450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.6.236.130:36069)
0925 01:23:03.285630 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.285635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.285652 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:03.285708 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.285711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.285772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.285854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.287061 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.287799 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.288968 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.289351 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.289440 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.293227 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.293242 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.293258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.293291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.295817 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.296408 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.296415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.296417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.296535 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.296880 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.296886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.296890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.297277 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.297613 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.297853 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.298074 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:23:03.299237 7982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:03.388823 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7859
I20260925 01:23:03.402751 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7992
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[7619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:03.416833 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:03.403509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.6.236.131:33259)
0925 01:23:03.403950 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:03.403954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:03.403970 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:03.404024 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:03.404027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:03.404086 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:03.404172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:03.405388 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.406087 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.407446 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:03.407837 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:03.407938 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:03.411793 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:03.411808 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:03.411824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:03.411864 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:03.414522 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.415091 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.415098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.415101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.415235 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:03.415527 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:03.415531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:03.415535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:03.415921 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:03.416152 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:03.416337 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:23:03.416492 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260925 01:23:03.417443 8115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:03.503938 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7992
I20260925 01:23:03.517733 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 7632
I20260925 01:23:03.518880 7694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:03.625859 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 7632
2026-09-25T01:23:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4003 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[8168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[8168](info): set up 2 sockets
Sep 25 01:23:03 dist-test-slave-s0b2 krb5kdc[8168](info): commencing operation
krb5kdc: starting...
W20260925 01:23:04.603123 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:04 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:04.763109 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:41495
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:41495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:05.064831 8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:05.065093 8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:05.065220 8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:05.072137 8181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:05.072232 8181 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:23:05.072276 8181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:05.072361 8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:05.072424 8181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:05.072513 8181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:05.081949 8181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:41495
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:41495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:05.083246 8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:05.084445 8181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:05.095649 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:05.095731 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:05.095628 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:05.096617 8181 server_base.cc:1034] running on GCE node
I20260925 01:23:05.097373 8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:05.098704 8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:05.099931 8181 hybrid_clock.cc:648] HybridClock initialized: now 1790299385099864 us; error 40 us; skew 500 ppm
I20260925 01:23:05.100238 8181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:05 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:05.103161 8181 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:05.104425 8181 webserver.cc:460] Webserver started at http://127.6.236.190:43489/ using document root <none> and password file <none>
I20260925 01:23:05.105075 8181 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:05.105171 8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:05.105458 8181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:05.108263 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "f08ec27cab1349ab8674d0fa9d5e80b6"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "63b0e2ddb506fe6ae2abe95ecda472c0"
server_key_iv: "a06d909744f4aaed74e9fd01140b37f7"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.108979 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "f08ec27cab1349ab8674d0fa9d5e80b6"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "63b0e2ddb506fe6ae2abe95ecda472c0"
server_key_iv: "a06d909744f4aaed74e9fd01140b37f7"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.113787 8181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 01:23:05.117146 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:05.118423 8181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:05.118570 8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "f08ec27cab1349ab8674d0fa9d5e80b6"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "63b0e2ddb506fe6ae2abe95ecda472c0"
server_key_iv: "a06d909744f4aaed74e9fd01140b37f7"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.118755 8181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:05.139858 8181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:05.144105 8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:05.144357 8181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:05.163295 8181 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:41495
I20260925 01:23:05.163314 8257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:41495 every 8 connection(s)
I20260925 01:23:05.164606 8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:05.167162 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8181
I20260925 01:23:05.167395 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:05.168200 7090 external_mini_cluster.cc:1428] Setting key 499ac8f79f2cd440c881c374e78e58ea
I20260925 01:23:05.168829 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:05.182317 8258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Bootstrap starting.
Sep 25 01:23:05 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:05.185910 8258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:05.187258 8258 log.cc:824] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:05.192421 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:05.170183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.6.236.190:41495)
0925 01:23:05.170949 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:05.170961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:05.171000 (+ 39us) server_negotiation.cc:407] Connection header received
0925 01:23:05.173234 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:05.173257 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:05.173654 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:05.173974 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:05.175690 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.176957 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.178606 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.179290 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.179427 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:05.183845 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:05.183855 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:05.183873 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:05.183928 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:05.187587 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.188488 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.188497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.188506 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.188728 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.189187 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.189192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.189196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.189921 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:05.190273 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:05.190713 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 01:23:05.191134 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":339,"threads_started":1}
I20260925 01:23:05.192592 8258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: No bootstrap required, opened a new log
I20260925 01:23:05.200347 8258 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } }
I20260925 01:23:05.200700 8258 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:05.200795 8258 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08ec27cab1349ab8674d0fa9d5e80b6, State: Initialized, Role: FOLLOWER
I20260925 01:23:05.201406 8258 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } }
I20260925 01:23:05.201659 8258 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:05.201794 8258 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:05.201944 8258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:05.203982 8258 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } }
I20260925 01:23:05.204442 8258 leader_election.cc:302] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08ec27cab1349ab8674d0fa9d5e80b6; no voters:
I20260925 01:23:05.205004 8258 leader_election.cc:288] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:05.205129 8263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:05.205770 8263 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [term 1 LEADER]: Becoming Leader. State: Replica: f08ec27cab1349ab8674d0fa9d5e80b6, State: Running, Role: LEADER
I20260925 01:23:05.206328 8263 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } }
I20260925 01:23:05.206884 8258 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:05.208961 8264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } } }
I20260925 01:23:05.209034 8265 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08ec27cab1349ab8674d0fa9d5e80b6. Latest consensus state: current_term: 1 leader_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08ec27cab1349ab8674d0fa9d5e80b6" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 41495 } } }
I20260925 01:23:05.209306 8264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:05.209342 8265 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:05.210166 8270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:05.215838 8270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:05.224110 8270 catalog_manager.cc:1347] Generated new cluster ID: 414551dead3d40b9b33efad86da172f9
I20260925 01:23:05.224215 8270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:05.244132 8270 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:05.245523 8270 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:05.252251 8270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Generated new TSK 0
I20260925 01:23:05.253273 8270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:05.303527 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41495
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:05.611580 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:05.611856 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:05.611979 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:05.619232 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:05.619338 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:05.619454 8285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:05.619555 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:05.629281 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:41495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:05.630595 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:05.632112 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:05.644498 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:05.644469 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:05.644572 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:05.645380 8285 server_base.cc:1034] running on GCE node
I20260925 01:23:05.645949 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:05.646867 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:05.648083 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790299385648020 us; error 38 us; skew 500 ppm
I20260925 01:23:05.648401 8285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:05 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:05.651464 8285 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:05.652880 8285 webserver.cc:460] Webserver started at http://127.6.236.129:36779/ using document root <none> and password file <none>
I20260925 01:23:05.653576 8285 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:05.653676 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:05.653988 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:05.656886 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "5f60810004e943298bfbe88a0589ee3f"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "801cc8a6d8bf687f2d1ae1ddce54f0cf"
server_key_iv: "49a152d960ae5c0908449c1d0aeaf6f3"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.657660 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "5f60810004e943298bfbe88a0589ee3f"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "801cc8a6d8bf687f2d1ae1ddce54f0cf"
server_key_iv: "49a152d960ae5c0908449c1d0aeaf6f3"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.662712 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 01:23:05.666008 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:05.667397 8285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:05.667551 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "5f60810004e943298bfbe88a0589ee3f"
format_stamp: "Formatted at 2026-09-25 01:23:05 on dist-test-slave-s0b2"
server_key: "801cc8a6d8bf687f2d1ae1ddce54f0cf"
server_key_iv: "49a152d960ae5c0908449c1d0aeaf6f3"
server_key_version: "encryptionkey@0"
I20260925 01:23:05.667745 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:05.687767 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:05.691820 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:05.692086 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:05.694653 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:05.694738 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:05.694873 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:05.694939 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:05 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:05.718928 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:05.695244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58745 (local address 127.6.236.190:41495)
0925 01:23:05.697812 (+ 2568us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:05.697816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:05.697835 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:05.700096 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:05.700102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:05.700196 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:05.700320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:05.702744 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.703717 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.706523 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.707059 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.707194 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:05.711734 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:05.711741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:05.711746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:05.711791 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:05.714969 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.715989 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.715994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.715998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.716149 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.716744 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.716750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.716753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.717091 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:05.717240 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:05.718322 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0925 01:23:05.718555 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2461,"thread_start_us":2359,"threads_started":1}
I20260925 01:23:05.719954 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:05.695995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.129:58745)
0925 01:23:05.696983 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:05.697041 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:05.699742 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:05.700634 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:05.700645 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:05.701523 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:05.702538 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:05.702566 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.703871 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.703877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:05.706354 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:05.706370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.707462 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.707473 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:05.707787 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:05.708582 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:05.708622 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:05.711459 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:05.715111 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:05.715121 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:05.715140 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:05.715841 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:05.716260 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:05.716269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:05.716274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:05.716587 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:05.717323 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:05.717334 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:05.717839 (+ 505us) client_negotiation.cc:769] Sending connection context
0925 01:23:05.718193 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 01:23:05.718761 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":423,"threads_started":1}
I20260925 01:23:05.737969 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:05.725296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58013 (local address 127.6.236.190:41495)
0925 01:23:05.725487 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:05.725494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:05.725911 (+ 417us) server_negotiation.cc:407] Connection header received
0925 01:23:05.726046 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:05.726051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:05.726133 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:05.726265 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:05.727093 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.728216 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.729720 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.730388 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.730523 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:05.732118 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:05.732126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:05.732131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:05.732171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:05.735316 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.736033 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.736039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.736043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.736192 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:05.736523 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:05.736529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:05.736532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:05.736863 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:05.737021 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:05.737418 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 01:23:05.737624 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:05.738056 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:05.725250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.129:58013)
0925 01:23:05.725829 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:05.725847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:05.725953 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:05.726290 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:05.726293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:05.726703 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:05.726975 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:05.726984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.728340 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.728343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:05.729564 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:05.729585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:05.730591 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:05.730596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:05.730706 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:05.731293 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:05.731309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:05.731947 (+ 638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:05.735453 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:05.735459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:05.735462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:05.735916 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:05.736292 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:05.736295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:05.736297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:05.736413 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:05.736990 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:05.736993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:05.737152 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 01:23:05.737309 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:23:05.737621 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
I20260925 01:23:05.744488 8285 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:44585
I20260925 01:23:05.744529 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:44585 every 8 connection(s)
I20260925 01:23:05.745605 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:05.747460 8421 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41495
I20260925 01:23:05.747646 8421 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:05.748189 8421 heartbeater.cc:511] Master 127.6.236.190:41495 requested a full tablet report, sending...
I20260925 01:23:05.749972 8213 ts_manager.cc:194] Registered new tserver with Master: 5f60810004e943298bfbe88a0589ee3f (127.6.236.129:44585)
I20260925 01:23:05.750840 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8285
I20260925 01:23:05.751030 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:05.751578 8213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:58745
I20260925 01:23:05.751768 7090 external_mini_cluster.cc:1428] Setting key aa36e28cf29542550730cbf7e47edae5
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:05.786435 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41495
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:06.091221 8425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:06.091485 8425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:06.091616 8425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:06.098574 8425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:06.098675 8425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:06.098783 8425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:06.098881 8425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:06.108618 8425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:41495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:06.109880 8425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:06.111156 8425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:06.123255 8430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:06.123209 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:06.123653 8433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:06.124295 8425 server_base.cc:1034] running on GCE node
I20260925 01:23:06.124830 8425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:06.125620 8425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:06.126818 8425 hybrid_clock.cc:648] HybridClock initialized: now 1790299386126754 us; error 38 us; skew 500 ppm
I20260925 01:23:06.127070 8425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:06.129956 8425 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:06.131246 8425 webserver.cc:460] Webserver started at http://127.6.236.130:41575/ using document root <none> and password file <none>
I20260925 01:23:06.131876 8425 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:06.131968 8425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:06.132258 8425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:06.135109 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "b0d200ace75a4d4e9d931a6989ad142b"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "06e58851ca2b78eeb73f0e388c139c66"
server_key_iv: "a9566b14e89c1317c4eff4c43435d468"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.135845 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d200ace75a4d4e9d931a6989ad142b"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "06e58851ca2b78eeb73f0e388c139c66"
server_key_iv: "a9566b14e89c1317c4eff4c43435d468"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.140646 8425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:23:06.143877 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:06.145107 8425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 01:23:06.145248 8425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "b0d200ace75a4d4e9d931a6989ad142b"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "06e58851ca2b78eeb73f0e388c139c66"
server_key_iv: "a9566b14e89c1317c4eff4c43435d468"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.145430 8425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:06.162516 8425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:06.167578 8425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:06.167833 8425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:06.170923 8425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:06.171059 8425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:06.171196 8425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:06.171267 8425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:06.192008 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.171149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:37043 (local address 127.6.236.190:41495)
0925 01:23:06.171389 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.171396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.172816 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 01:23:06.175079 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.175086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.175160 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.175218 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:06.177665 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.178838 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.181661 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.182276 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.182400 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.186175 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:06.186181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:06.186184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:06.186218 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:06.188665 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.189354 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.189359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.189361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.189489 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.189923 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.189928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.189931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.190205 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:06.190456 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.191380 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.191624 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 01:23:06.193151 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.171801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.130:37043)
0925 01:23:06.172659 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.172707 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.174670 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.175728 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.175739 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.176468 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:06.177427 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.177459 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.179027 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.179032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.181530 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.181549 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.182543 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.182550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.182752 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.183368 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:06.183390 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:06.185955 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:06.188794 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.188803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.188819 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.189241 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.189609 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.189614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.189618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.189817 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.190383 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:06.190390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:06.190884 (+ 494us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.191214 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.191701 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260925 01:23:06.212137 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.199975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:53103 (local address 127.6.236.190:41495)
0925 01:23:06.200135 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.200141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.200735 (+ 594us) server_negotiation.cc:407] Connection header received
0925 01:23:06.200894 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.200899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.200958 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.201031 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:06.201902 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.202745 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.203985 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.204390 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.204494 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.206069 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:06.206074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:06.206077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:06.206113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:06.209045 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.209823 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.209829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.209832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.209996 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.210518 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.210523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.210527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.210961 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:06.211121 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.211611 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.211817 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:23:06.212307 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.200080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.130:53103)
0925 01:23:06.200646 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.200663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.200782 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.201067 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.201071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.201503 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:06.201773 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.201782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.202864 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.202869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.203858 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.203869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.204555 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.204562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.204722 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.205243 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:06.205259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:06.205890 (+ 631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:06.209168 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.209173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.209177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.209718 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.210136 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.210141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.210145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.210341 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.211130 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:06.211135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:06.211365 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.211497 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.211835 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":275,"threads_started":1}
I20260925 01:23:06.219775 8425 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:32825
I20260925 01:23:06.219796 8559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:32825 every 8 connection(s)
I20260925 01:23:06.220866 8425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:06.221122 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8425
I20260925 01:23:06.221288 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:06.221832 7090 external_mini_cluster.cc:1428] Setting key 2ccfa27be00152c49d152412a639b64c
I20260925 01:23:06.222939 8560 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41495
I20260925 01:23:06.223119 8560 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:06.223750 8560 heartbeater.cc:511] Master 127.6.236.190:41495 requested a full tablet report, sending...
I20260925 01:23:06.224933 8213 ts_manager.cc:194] Registered new tserver with Master: b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825)
I20260925 01:23:06.225670 8213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:37043
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:06.257252 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:41495
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:06.560236 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:06.560506 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:06.560616 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:06.567556 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:06.567659 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:06.567749 8564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:06.567826 8564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:06.577309 8564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:41495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:06.578503 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:06.579754 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:06.591614 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:06.591670 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:06.591681 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:06.592577 8564 server_base.cc:1034] running on GCE node
I20260925 01:23:06.593124 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:06.593941 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:06.595145 8564 hybrid_clock.cc:648] HybridClock initialized: now 1790299386595080 us; error 53 us; skew 500 ppm
I20260925 01:23:06.595410 8564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:06.598315 8564 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:06.599627 8564 webserver.cc:460] Webserver started at http://127.6.236.131:37725/ using document root <none> and password file <none>
I20260925 01:23:06.600267 8564 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:06.600360 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:06.600647 8564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:06.603505 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "225a5bba1cea47f0bb41970cb9259f40"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "70665809821fab8d92e3b1cc0079a393"
server_key_iv: "7177e03992f102c1efee5a6329a58de7"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.604241 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "225a5bba1cea47f0bb41970cb9259f40"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "70665809821fab8d92e3b1cc0079a393"
server_key_iv: "7177e03992f102c1efee5a6329a58de7"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.609185 8564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:23:06.612543 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:06.613909 8564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:06.614032 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "225a5bba1cea47f0bb41970cb9259f40"
format_stamp: "Formatted at 2026-09-25 01:23:06 on dist-test-slave-s0b2"
server_key: "70665809821fab8d92e3b1cc0079a393"
server_key_iv: "7177e03992f102c1efee5a6329a58de7"
server_key_version: "encryptionkey@0"
I20260925 01:23:06.614205 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:06.649729 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:06.653707 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:06.653960 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:06.657855 8564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:06.657969 8564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:06.658104 8564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:06.658169 8564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:06.686360 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.659909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:51135 (local address 127.6.236.190:41495)
0925 01:23:06.660415 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.660422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.663300 (+ 2878us) server_negotiation.cc:407] Connection header received
0925 01:23:06.666619 (+ 3319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.666626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.666730 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.666808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:06.669603 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.670545 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.672899 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.673337 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.673456 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.678603 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:06.678611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:06.678616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:06.678661 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:06.682207 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.683259 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.683266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.683269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.683440 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.684010 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.684015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.684018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.684412 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:06.684566 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.685642 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.685924 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260925 01:23:06.687727 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.659895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.131:51135)
0925 01:23:06.663132 (+ 3237us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.663188 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.666205 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.667499 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.667511 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.668303 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:06.669408 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.669437 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.670744 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.670748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.672752 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.672765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.673568 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.673576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.673823 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.674850 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:06.674882 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:06.678273 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:06.682440 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.682451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.682473 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.683124 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.683571 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.683577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.683581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.683845 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.684665 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:06.684675 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:06.685172 (+ 497us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.685492 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.686062 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2741,"thread_start_us":2394,"threads_started":1}
I20260925 01:23:06.711843 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.694961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.6.236.131:57753)
0925 01:23:06.695579 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.695596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.695708 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.696079 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.696084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.696657 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:06.697039 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.697053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.698527 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.698533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.700276 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.700294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.701741 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.701749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.701942 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.702634 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:06.702664 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:06.704064 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:06.708109 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.708114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.708119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.708768 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.709216 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:06.709221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:06.709225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:06.709408 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:06.710160 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:06.710165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:06.710466 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.710644 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.711028 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":325,"threads_started":1}
I20260925 01:23:06.711858 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.694914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57753 (local address 127.6.236.190:41495)
0925 01:23:06.695126 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.695133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.695660 (+ 527us) server_negotiation.cc:407] Connection header received
0925 01:23:06.695815 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.695821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.695917 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.696025 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:06.697198 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.698390 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.700503 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.701522 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.701681 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.704313 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:06.704321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:06.704325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:06.704366 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:06.707926 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.708933 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.708939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.708943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.709101 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.709556 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.709561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.709564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.709936 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:06.710092 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.711219 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.711454 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:23:06.718269 8564 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44281
I20260925 01:23:06.718322 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44281 every 8 connection(s)
I20260925 01:23:06.719449 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:06.721278 8699 heartbeater.cc:347] Connected to a master server at 127.6.236.190:41495
I20260925 01:23:06.721467 8699 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:06.722091 8699 heartbeater.cc:511] Master 127.6.236.190:41495 requested a full tablet report, sending...
I20260925 01:23:06.723431 8213 ts_manager.cc:194] Registered new tserver with Master: 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281)
I20260925 01:23:06.724047 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8564
I20260925 01:23:06.724220 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:06.724239 8213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:51135
I20260925 01:23:06.724726 7090 external_mini_cluster.cc:1428] Setting key 5a4c7223a83581a7b8c99be62a5389b9
I20260925 01:23:06.727684 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 01:23:06.754477 8421 heartbeater.cc:503] Master 127.6.236.190:41495 was elected leader, sending a full tablet report...
Sep 25 01:23:06 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:06.768491 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.756006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.6.236.190:41495)
0925 01:23:06.756181 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.756187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.756636 (+ 449us) server_negotiation.cc:407] Connection header received
0925 01:23:06.756776 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.756780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.756846 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.756916 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:06.758169 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.758914 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.760071 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.760481 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.760567 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.763942 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:06.763947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:06.763950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:06.763981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:06.766406 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.766910 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.766914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.766917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.767029 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:06.767301 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:06.767304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:06.767306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:06.767562 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:06.767733 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.767963 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.768133 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:23:06.772187 8213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:06.779781 8213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:06.813758 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.791202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44281 (local address 127.0.0.1:46198)
0925 01:23:06.803100 (+ 11898us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.803142 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.803329 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.805053 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.805060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.805090 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:06.805495 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.805511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.807801 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.807807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.810497 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.810515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.812791 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.812796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.812978 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.813032 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.813167 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.813366 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11663,"thread_start_us":11506,"threads_started":1}
I20260925 01:23:06.813752 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.791311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.6.236.131:44281)
0925 01:23:06.802525 (+ 11214us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.802535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.803617 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 01:23:06.803759 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.803768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.804049 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.804228 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:06.805773 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.807632 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.810708 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.812242 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.812356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.812620 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.813232 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.813404 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10779,"thread_start_us":10933,"threads_started":1}
I20260925 01:23:06.815289 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.791825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.6.236.130:32825)
0925 01:23:06.807270 (+ 15445us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.807279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.807309 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:23:06.807404 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.807412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.807653 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.807844 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:06.808884 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.810478 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.812927 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.814275 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.814408 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.814610 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.814818 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.814979 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15046,"thread_start_us":15068,"threads_started":1}
I20260925 01:23:06.817955 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86296b54d442459630603381bfbc69 (DEFAULT_TABLE table=test-table [id=da47456146ed45419b563ab5d9ddf5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:06.819414 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86296b54d442459630603381bfbc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:06.820045 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.805001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:44585 (local address 127.0.0.1:40278)
0925 01:23:06.811763 (+ 6762us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.811785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.811901 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.813099 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.813102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.813120 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:06.813464 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.813485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.815681 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.815686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.817617 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.817633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.819184 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.819191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.819353 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.819386 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.819589 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.819715 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6639,"thread_start_us":6488,"threads_started":1}
I20260925 01:23:06.820742 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.805296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.6.236.129:44585)
0925 01:23:06.812338 (+ 7042us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.812346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.812372 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:23:06.812458 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.812465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.812750 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.812918 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:06.813680 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.815535 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.817759 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.818978 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.819097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.819235 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.820304 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.820466 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6890,"thread_start_us":6840,"threads_started":1}
I20260925 01:23:06.824007 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86296b54d442459630603381bfbc69 (DEFAULT_TABLE table=test-table [id=da47456146ed45419b563ab5d9ddf5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:06.825338 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86296b54d442459630603381bfbc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:06.826512 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.791333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:32825 (local address 127.0.0.1:59484)
0925 01:23:06.804784 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.804801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.804892 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.808052 (+ 3160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.808058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.808088 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:06.808687 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.808700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.810652 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.810658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.812769 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.812783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.814463 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.814470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.814646 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.814675 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.814752 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.814872 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13344,"thread_start_us":13241,"threads_started":1}
I20260925 01:23:06.830210 8490 tablet_service.cc:1467] Processing CreateTablet for tablet 4e86296b54d442459630603381bfbc69 (DEFAULT_TABLE table=test-table [id=da47456146ed45419b563ab5d9ddf5fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:06.831728 8490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e86296b54d442459630603381bfbc69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:06.832213 8718 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Bootstrap starting.
I20260925 01:23:06.835363 8718 tablet_bootstrap.cc:654] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:06.836354 8718 log.cc:824] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:06.839545 8720 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Bootstrap starting.
I20260925 01:23:06.843182 8720 tablet_bootstrap.cc:654] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:06.844386 8720 log.cc:824] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:06.844930 8718 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: No bootstrap required, opened a new log
I20260925 01:23:06.845222 8718 ts_tablet_manager.cc:1397] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 01:23:06.845383 8721 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Bootstrap starting.
I20260925 01:23:06.848465 8721 tablet_bootstrap.cc:654] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:06.848943 8720 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: No bootstrap required, opened a new log
I20260925 01:23:06.849212 8720 ts_tablet_manager.cc:1397] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:23:06.849596 8721 log.cc:824] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:06.852167 8721 tablet_bootstrap.cc:492] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: No bootstrap required, opened a new log
I20260925 01:23:06.852409 8721 ts_tablet_manager.cc:1397] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:23:06.855181 8718 raft_consensus.cc:348] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.855631 8718 raft_consensus.cc:374] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:06.855727 8718 raft_consensus.cc:729] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225a5bba1cea47f0bb41970cb9259f40, State: Initialized, Role: FOLLOWER
I20260925 01:23:06.856333 8718 consensus_queue.cc:260] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.857425 8699 heartbeater.cc:503] Master 127.6.236.190:41495 was elected leader, sending a full tablet report...
I20260925 01:23:06.858177 8718 ts_tablet_manager.cc:1428] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260925 01:23:06.860289 8721 raft_consensus.cc:348] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.860625 8721 raft_consensus.cc:374] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:06.860375 8720 raft_consensus.cc:348] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.860718 8721 raft_consensus.cc:729] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d200ace75a4d4e9d931a6989ad142b, State: Initialized, Role: FOLLOWER
I20260925 01:23:06.860775 8720 raft_consensus.cc:374] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:06.860881 8720 raft_consensus.cc:729] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f60810004e943298bfbe88a0589ee3f, State: Initialized, Role: FOLLOWER
I20260925 01:23:06.861302 8721 consensus_queue.cc:260] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.861455 8720 consensus_queue.cc:260] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.862524 8560 heartbeater.cc:503] Master 127.6.236.190:41495 was elected leader, sending a full tablet report...
I20260925 01:23:06.864214 8721 ts_tablet_manager.cc:1428] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260925 01:23:06.865664 8720 ts_tablet_manager.cc:1428] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260925 01:23:06.945024 8726 raft_consensus.cc:482] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:06.945338 8726 raft_consensus.cc:504] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.947081 8726 leader_election.cc:288] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281), b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825)
I20260925 01:23:06.953362 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.947607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58801 (local address 127.6.236.130:32825)
0925 01:23:06.947817 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.947823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.948241 (+ 418us) server_negotiation.cc:407] Connection header received
0925 01:23:06.948448 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.948453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.948517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.948597 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:06.949138 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.949901 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.951762 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.952488 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.952644 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.952730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.953008 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.953130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 01:23:06.953470 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.947462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:51121 (local address 127.6.236.131:44281)
0925 01:23:06.947735 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:06.947743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:06.948373 (+ 630us) server_negotiation.cc:407] Connection header received
0925 01:23:06.948546 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:06.948549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:06.948606 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:06.948677 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:06.949331 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.950085 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.951801 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.952584 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.952745 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.952817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:06.953098 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 01:23:06.953235 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 01:23:06.953747 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.947398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:32825 (local address 127.6.236.129:58801)
0925 01:23:06.948139 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.948160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.948302 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.948623 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.948628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.948651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:06.949010 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.949020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.950026 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.950031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.951613 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.951629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.952680 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.952686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.952864 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.952889 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.952967 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.953093 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":43,"thread_start_us":349,"threads_started":1}
I20260925 01:23:06.953790 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:06.947425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44281 (local address 127.6.236.129:51121)
0925 01:23:06.948280 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:06.948296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:06.948426 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:06.948703 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:06.948708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:06.948766 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:06.949182 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.949192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.950221 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.950226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.951659 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:06.951672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:06.952804 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:06.952810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:06.952960 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:06.952982 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:23:06.953059 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:23:06.953179 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":16,"thread_start_us":543,"threads_started":1}
I20260925 01:23:06.954772 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86296b54d442459630603381bfbc69" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d200ace75a4d4e9d931a6989ad142b" is_pre_election: true
I20260925 01:23:06.954773 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86296b54d442459630603381bfbc69" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40" is_pre_election: true
I20260925 01:23:06.955323 8654 raft_consensus.cc:2454] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 0.
I20260925 01:23:06.955323 8514 raft_consensus.cc:2454] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 0.
I20260925 01:23:06.955938 8301 leader_election.cc:302] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225a5bba1cea47f0bb41970cb9259f40, 5f60810004e943298bfbe88a0589ee3f; no voters:
I20260925 01:23:06.956264 8726 raft_consensus.cc:2781] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:06.956445 8726 raft_consensus.cc:482] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:06.956543 8726 raft_consensus.cc:3037] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:06.958753 8726 raft_consensus.cc:504] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.959275 8726 leader_election.cc:288] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 election: Requested vote from peers 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281), b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825)
I20260925 01:23:06.959689 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86296b54d442459630603381bfbc69" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40"
I20260925 01:23:06.959733 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e86296b54d442459630603381bfbc69" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d200ace75a4d4e9d931a6989ad142b"
I20260925 01:23:06.959882 8654 raft_consensus.cc:3037] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:06.959939 8514 raft_consensus.cc:3037] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:06.962124 8654 raft_consensus.cc:2454] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 1.
I20260925 01:23:06.962133 8514 raft_consensus.cc:2454] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 1.
I20260925 01:23:06.962494 8301 leader_election.cc:302] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225a5bba1cea47f0bb41970cb9259f40, 5f60810004e943298bfbe88a0589ee3f; no voters:
I20260925 01:23:06.962773 8726 raft_consensus.cc:2781] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:06.963148 8726 raft_consensus.cc:686] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 1 LEADER]: Becoming Leader. State: Replica: 5f60810004e943298bfbe88a0589ee3f, State: Running, Role: LEADER
I20260925 01:23:06.963567 8726 consensus_queue.cc:237] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } }
I20260925 01:23:06.968019 8212 catalog_manager.cc:5495] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f60810004e943298bfbe88a0589ee3f (127.6.236.129). New cstate: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } health_report { overall_health: HEALTHY } } }
W20260925 01:23:06.970849 8700 tablet.cc:2369] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:23:06.972864 8561 tablet.cc:2369] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:23:06.997876 8422 tablet.cc:2369] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:07.014999 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.009510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.6.236.129:44585)
0925 01:23:07.009730 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.009736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.009861 (+ 125us) server_negotiation.cc:407] Connection header received
0925 01:23:07.010170 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.010175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.010312 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.010444 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:07.010808 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.011499 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.012872 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.013296 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.013403 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.013876 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.014415 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.014545 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.014666 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.014750 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:23:07.022686 8514 raft_consensus.cc:1261] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Refusing update from remote peer 5f60810004e943298bfbe88a0589ee3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.022663 8654 raft_consensus.cc:1261] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Refusing update from remote peer 5f60810004e943298bfbe88a0589ee3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.023653 8729 consensus_queue.cc:1035] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.024114 8726 consensus_queue.cc:1035] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.054957 8213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:07.055193 8213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:07.058475 8213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Sending DeleteTablet for 3 replicas of tablet 4e86296b54d442459630603381bfbc69
I20260925 01:23:07.059458 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86296b54d442459630603381bfbc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:46198
I20260925 01:23:07.059648 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86296b54d442459630603381bfbc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:40278
I20260925 01:23:07.059883 8490 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e86296b54d442459630603381bfbc69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:59484
I20260925 01:23:07.062458 8213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:07.063804 8213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:07.072971 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 (DEFAULT_TABLE table=test-table [id=56dcd916270e40beb1441db8cd4a94f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:07.073073 8489 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 (DEFAULT_TABLE table=test-table [id=56dcd916270e40beb1441db8cd4a94f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:07.073601 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca3e0e3305f459fbb109e4ade373170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.073603 8489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca3e0e3305f459fbb109e4ade373170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.079409 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 (DEFAULT_TABLE table=test-table [id=56dcd916270e40beb1441db8cd4a94f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:07.080019 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca3e0e3305f459fbb109e4ade373170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.082260 8721 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Bootstrap starting.
I20260925 01:23:07.084069 8721 tablet_bootstrap.cc:654] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.088681 8720 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Bootstrap starting.
I20260925 01:23:07.089304 8718 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Bootstrap starting.
I20260925 01:23:07.089862 8743 tablet_replica.cc:330] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: stopping tablet replica
I20260925 01:23:07.090222 8720 tablet_bootstrap.cc:654] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.090615 8743 raft_consensus.cc:2229] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:07.090849 8718 tablet_bootstrap.cc:654] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.091302 8743 raft_consensus.cc:2258] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.091923 8741 tablet_replica.cc:330] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: stopping tablet replica
I20260925 01:23:07.094312 8743 ts_tablet_manager.cc:1905] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.094785 8741 raft_consensus.cc:2229] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:07.095353 8741 raft_consensus.cc:2258] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.096426 8721 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: No bootstrap required, opened a new log
I20260925 01:23:07.096885 8721 ts_tablet_manager.cc:1397] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260925 01:23:07.097662 8721 raft_consensus.cc:348] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.097971 8721 raft_consensus.cc:374] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.098114 8721 raft_consensus.cc:729] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d200ace75a4d4e9d931a6989ad142b, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.098439 8721 consensus_queue.cc:260] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.099038 8721 ts_tablet_manager.cc:1428] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:23:07.100847 8741 ts_tablet_manager.cc:1905] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.105842 8743 ts_tablet_manager.cc:1918] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:07.106534 8742 tablet_replica.cc:330] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: stopping tablet replica
I20260925 01:23:07.107087 8742 raft_consensus.cc:2229] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:07.107967 8742 raft_consensus.cc:2258] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.108135 8720 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: No bootstrap required, opened a new log
I20260925 01:23:07.108296 8720 ts_tablet_manager.cc:1397] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260925 01:23:07.108909 8718 tablet_bootstrap.cc:492] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: No bootstrap required, opened a new log
I20260925 01:23:07.109052 8718 ts_tablet_manager.cc:1397] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260925 01:23:07.109184 8720 raft_consensus.cc:348] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.109400 8720 raft_consensus.cc:374] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.109498 8720 raft_consensus.cc:729] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f60810004e943298bfbe88a0589ee3f, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.109766 8720 consensus_queue.cc:260] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.109807 8718 raft_consensus.cc:348] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.109992 8718 raft_consensus.cc:374] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.110153 8718 raft_consensus.cc:729] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225a5bba1cea47f0bb41970cb9259f40, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.110348 8720 ts_tablet_manager.cc:1428] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 01:23:07.110481 8718 consensus_queue.cc:260] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.110996 8718 ts_tablet_manager.cc:1428] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:07.111589 8742 ts_tablet_manager.cc:1905] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.111809 8741 ts_tablet_manager.cc:1918] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:07.111966 8741 log.cc:1196] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/4e86296b54d442459630603381bfbc69
I20260925 01:23:07.112655 8741 ts_tablet_manager.cc:1939] T 4e86296b54d442459630603381bfbc69 P 225a5bba1cea47f0bb41970cb9259f40: Deleting consensus metadata
I20260925 01:23:07.114377 8197 catalog_manager.cc:4841] TS 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281): tablet 4e86296b54d442459630603381bfbc69 (table test-table [id=da47456146ed45419b563ab5d9ddf5fb]) successfully deleted
I20260925 01:23:07.115613 8743 log.cc:1196] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/4e86296b54d442459630603381bfbc69
I20260925 01:23:07.116747 8743 ts_tablet_manager.cc:1939] T 4e86296b54d442459630603381bfbc69 P b0d200ace75a4d4e9d931a6989ad142b: Deleting consensus metadata
I20260925 01:23:07.118462 8200 catalog_manager.cc:4841] TS b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825): tablet 4e86296b54d442459630603381bfbc69 (table test-table [id=da47456146ed45419b563ab5d9ddf5fb]) successfully deleted
I20260925 01:23:07.118664 8742 ts_tablet_manager.cc:1918] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:07.118783 8742 log.cc:1196] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/4e86296b54d442459630603381bfbc69
I20260925 01:23:07.119169 8742 ts_tablet_manager.cc:1939] T 4e86296b54d442459630603381bfbc69 P 5f60810004e943298bfbe88a0589ee3f: Deleting consensus metadata
I20260925 01:23:07.120373 8200 catalog_manager.cc:4841] TS 5f60810004e943298bfbe88a0589ee3f (127.6.236.129:44585): tablet 4e86296b54d442459630603381bfbc69 (table test-table [id=da47456146ed45419b563ab5d9ddf5fb]) successfully deleted
I20260925 01:23:07.199528 8725 raft_consensus.cc:482] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:07.199757 8725 raft_consensus.cc:504] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.201700 8725 leader_election.cc:288] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f60810004e943298bfbe88a0589ee3f (127.6.236.129:44585), 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281)
I20260925 01:23:07.210377 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.202040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:37283 (local address 127.6.236.131:44281)
0925 01:23:07.202228 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.202775 (+ 547us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.203646 (+ 871us) server_negotiation.cc:407] Connection header received
0925 01:23:07.203816 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.203821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.203907 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.203992 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:07.204616 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.205321 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.207913 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.209111 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.209235 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.209311 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.209916 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.210062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:23:07.210402 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.202560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:59897 (local address 127.6.236.129:44585)
0925 01:23:07.202743 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.202748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.203314 (+ 566us) server_negotiation.cc:407] Connection header received
0925 01:23:07.203623 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.203633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.203705 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.203786 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:07.204438 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.205601 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.208007 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.209218 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.209370 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.209438 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.209953 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.210081 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:23:07.210417 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.202170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44281 (local address 127.6.236.130:37283)
0925 01:23:07.203538 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.203555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.203674 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.204028 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.204032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.204053 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:07.204480 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.204490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.205447 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.205453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.207697 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.207712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.209514 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.209521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.209731 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.209766 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.209861 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.209992 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":4,"thread_start_us":1076,"threads_started":1}
I20260925 01:23:07.211021 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.201950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:44585 (local address 127.6.236.130:59897)
0925 01:23:07.203202 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.203224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.203441 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.203816 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.203820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.203844 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:07.204240 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.204252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.205740 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.205745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.207829 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.207842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.209611 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.209617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.209782 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.209806 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.210588 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.210712 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":967,"threads_started":1}
I20260925 01:23:07.211380 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca3e0e3305f459fbb109e4ade373170" candidate_uuid: "b0d200ace75a4d4e9d931a6989ad142b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40" is_pre_election: true
I20260925 01:23:07.211699 8654 raft_consensus.cc:2454] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d200ace75a4d4e9d931a6989ad142b in term 0.
I20260925 01:23:07.211891 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca3e0e3305f459fbb109e4ade373170" candidate_uuid: "b0d200ace75a4d4e9d931a6989ad142b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f60810004e943298bfbe88a0589ee3f" is_pre_election: true
I20260925 01:23:07.212182 8441 leader_election.cc:302] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225a5bba1cea47f0bb41970cb9259f40, b0d200ace75a4d4e9d931a6989ad142b; no voters:
I20260925 01:23:07.212327 8371 raft_consensus.cc:2454] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d200ace75a4d4e9d931a6989ad142b in term 0.
I20260925 01:23:07.212626 8725 raft_consensus.cc:2781] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:07.212741 8725 raft_consensus.cc:482] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:07.212826 8725 raft_consensus.cc:3037] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.215943 8725 raft_consensus.cc:504] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.216634 8725 leader_election.cc:288] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [CANDIDATE]: Term 1 election: Requested vote from peers 5f60810004e943298bfbe88a0589ee3f (127.6.236.129:44585), 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281)
I20260925 01:23:07.217105 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca3e0e3305f459fbb109e4ade373170" candidate_uuid: "b0d200ace75a4d4e9d931a6989ad142b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f60810004e943298bfbe88a0589ee3f"
I20260925 01:23:07.217082 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca3e0e3305f459fbb109e4ade373170" candidate_uuid: "b0d200ace75a4d4e9d931a6989ad142b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40"
I20260925 01:23:07.217280 8654 raft_consensus.cc:3037] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.217286 8371 raft_consensus.cc:3037] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.219990 8371 raft_consensus.cc:2454] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d200ace75a4d4e9d931a6989ad142b in term 1.
I20260925 01:23:07.220152 8654 raft_consensus.cc:2454] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d200ace75a4d4e9d931a6989ad142b in term 1.
I20260925 01:23:07.220403 8444 leader_election.cc:302] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f60810004e943298bfbe88a0589ee3f, b0d200ace75a4d4e9d931a6989ad142b; no voters:
I20260925 01:23:07.220731 8725 raft_consensus.cc:2781] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:07.221218 8725 raft_consensus.cc:686] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 1 LEADER]: Becoming Leader. State: Replica: b0d200ace75a4d4e9d931a6989ad142b, State: Running, Role: LEADER
I20260925 01:23:07.222059 8725 consensus_queue.cc:237] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.225899 8213 catalog_manager.cc:5495] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b reported cstate change: term changed from 0 to 1, leader changed from <none> to b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130). New cstate: current_term: 1 leader_uuid: "b0d200ace75a4d4e9d931a6989ad142b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:07.279162 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.264612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:41495 (local address 127.0.0.1:55866)
0925 01:23:07.265228 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.265247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.265380 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.265751 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.265755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.266489 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:07.266759 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.266769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.268169 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.268173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.269114 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.269125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.270099 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.270104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.270225 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.270913 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:07.270937 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:07.273351 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:07.276770 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:07.276781 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:07.276787 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:07.277191 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:07.277551 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:07.277554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:07.277561 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:07.277698 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:07.278263 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:07.278272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:07.278464 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.278605 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.278794 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260925 01:23:07.279323 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.264579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.6.236.190:41495)
0925 01:23:07.264802 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.264808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.265337 (+ 529us) server_negotiation.cc:407] Connection header received
0925 01:23:07.265492 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.265498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.265598 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.265714 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.266917 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.268031 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.269252 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.269889 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.270031 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.273547 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.273554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.273558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.273595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.276648 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.277295 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.277300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.277303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.277444 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.277787 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.277791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.277794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.278123 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.278277 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.278722 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.278932 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:23:07.283907 8213 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:55866:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:23:07.284847 8213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:07.291653 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 53d6054f30dd49a1b835f42eb5917275 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29c4bed0fcfd43cb924285fd12afd94d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:07.291668 8490 tablet_service.cc:1467] Processing CreateTablet for tablet 53d6054f30dd49a1b835f42eb5917275 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29c4bed0fcfd43cb924285fd12afd94d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:07.292233 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d6054f30dd49a1b835f42eb5917275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.292236 8490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d6054f30dd49a1b835f42eb5917275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.292146 8629 tablet_service.cc:1467] Processing CreateTablet for tablet 53d6054f30dd49a1b835f42eb5917275 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29c4bed0fcfd43cb924285fd12afd94d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:07.292691 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d6054f30dd49a1b835f42eb5917275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:07.299352 8721 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: Bootstrap starting.
I20260925 01:23:07.299675 8718 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: Bootstrap starting.
I20260925 01:23:07.300954 8721 tablet_bootstrap.cc:654] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.301259 8718 tablet_bootstrap.cc:654] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.304298 8720 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: Bootstrap starting.
I20260925 01:23:07.306005 8720 tablet_bootstrap.cc:654] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:07.310696 8718 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: No bootstrap required, opened a new log
I20260925 01:23:07.310703 8720 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: No bootstrap required, opened a new log
I20260925 01:23:07.310856 8718 ts_tablet_manager.cc:1397] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260925 01:23:07.310871 8720 ts_tablet_manager.cc:1397] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:23:07.310941 8721 tablet_bootstrap.cc:492] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: No bootstrap required, opened a new log
I20260925 01:23:07.311082 8721 ts_tablet_manager.cc:1397] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260925 01:23:07.311578 8718 raft_consensus.cc:348] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.311631 8720 raft_consensus.cc:348] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.311863 8718 raft_consensus.cc:374] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.311761 8721 raft_consensus.cc:348] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.311883 8720 raft_consensus.cc:374] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.311973 8721 raft_consensus.cc:374] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:07.311973 8720 raft_consensus.cc:729] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f60810004e943298bfbe88a0589ee3f, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.312037 8721 raft_consensus.cc:729] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d200ace75a4d4e9d931a6989ad142b, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.312216 8720 consensus_queue.cc:260] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.312283 8721 consensus_queue.cc:260] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.312781 8721 ts_tablet_manager.cc:1428] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 01:23:07.312806 8720 ts_tablet_manager.cc:1428] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:07.311954 8718 raft_consensus.cc:729] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225a5bba1cea47f0bb41970cb9259f40, State: Initialized, Role: FOLLOWER
I20260925 01:23:07.313171 8718 consensus_queue.cc:260] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.313691 8718 ts_tablet_manager.cc:1428] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:07.313808 8724 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.314144 8724 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.314189 8726 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.314483 8726 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.314759 8725 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.315018 8725 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.366693 8726 raft_consensus.cc:482] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:07.366874 8726 raft_consensus.cc:504] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.367533 8726 leader_election.cc:288] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825), 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281)
I20260925 01:23:07.368067 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d6054f30dd49a1b835f42eb5917275" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40" is_pre_election: true
I20260925 01:23:07.368103 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d6054f30dd49a1b835f42eb5917275" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d200ace75a4d4e9d931a6989ad142b" is_pre_election: true
I20260925 01:23:07.368319 8654 raft_consensus.cc:2454] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 0.
I20260925 01:23:07.368386 8514 raft_consensus.cc:2454] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 0.
I20260925 01:23:07.368796 8301 leader_election.cc:302] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225a5bba1cea47f0bb41970cb9259f40, 5f60810004e943298bfbe88a0589ee3f; no voters:
I20260925 01:23:07.369119 8726 raft_consensus.cc:2781] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:07.369239 8726 raft_consensus.cc:482] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:07.369333 8726 raft_consensus.cc:3037] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.371578 8726 raft_consensus.cc:504] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.372581 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d6054f30dd49a1b835f42eb5917275" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d200ace75a4d4e9d931a6989ad142b"
I20260925 01:23:07.372558 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d6054f30dd49a1b835f42eb5917275" candidate_uuid: "5f60810004e943298bfbe88a0589ee3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225a5bba1cea47f0bb41970cb9259f40"
I20260925 01:23:07.372780 8514 raft_consensus.cc:3037] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.372798 8654 raft_consensus.cc:3037] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:07.375036 8726 leader_election.cc:288] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 election: Requested vote from peers b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825), 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281)
I20260925 01:23:07.375590 8514 raft_consensus.cc:2454] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 1.
I20260925 01:23:07.375571 8654 raft_consensus.cc:2454] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f60810004e943298bfbe88a0589ee3f in term 1.
I20260925 01:23:07.376034 8304 leader_election.cc:302] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f60810004e943298bfbe88a0589ee3f, b0d200ace75a4d4e9d931a6989ad142b; no voters:
I20260925 01:23:07.376366 8726 raft_consensus.cc:2781] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:07.376529 8726 raft_consensus.cc:686] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [term 1 LEADER]: Becoming Leader. State: Replica: 5f60810004e943298bfbe88a0589ee3f, State: Running, Role: LEADER
I20260925 01:23:07.376834 8726 consensus_queue.cc:237] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } }
I20260925 01:23:07.378666 8729 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: TxnStatusTablet state changed. Reason: New leader 5f60810004e943298bfbe88a0589ee3f. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.378994 8729 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:07.379655 8761 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:23:07.379855 8761 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:23:07.380214 8213 catalog_manager.cc:5495] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f60810004e943298bfbe88a0589ee3f (127.6.236.129). New cstate: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:07.406541 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.401379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:44585 (local address 127.0.0.1:40290)
0925 01:23:07.401656 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.401673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.401810 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.402176 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.402181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.402193 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:07.402389 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.402398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.403442 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.403447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.404448 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.404463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.405463 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.405467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.405611 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.405665 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.406100 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.406116 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.406170 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.406315 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 01:23:07.406654 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.401391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.6.236.129:44585)
0925 01:23:07.401603 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.401608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.401733 (+ 125us) server_negotiation.cc:407] Connection header received
0925 01:23:07.401984 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.401988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.402056 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.402135 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:07.402523 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.403321 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.404643 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.405310 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.405462 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.405869 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.406012 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.406108 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.406230 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.406347 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:23:07.409864 8514 raft_consensus.cc:1261] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Refusing update from remote peer 5f60810004e943298bfbe88a0589ee3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.410194 8654 raft_consensus.cc:1261] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Refusing update from remote peer 5f60810004e943298bfbe88a0589ee3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.410465 8726 consensus_queue.cc:1035] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.410815 8729 consensus_queue.cc:1035] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.414104 8725 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: TxnStatusTablet state changed. Reason: New leader 5f60810004e943298bfbe88a0589ee3f. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.414390 8725 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.415712 8724 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: TxnStatusTablet state changed. Reason: New leader 5f60810004e943298bfbe88a0589ee3f. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.415961 8724 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.417820 8729 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.418067 8729 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:07.419736 8724 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.419951 8724 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 225a5bba1cea47f0bb41970cb9259f40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.420433 8726 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.420709 8726 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P 5f60810004e943298bfbe88a0589ee3f: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:07.421736 8734 mvcc.cc:205] Tried to move back new op lower bound from 7333066290825621504 to 7333066290698334208. Current Snapshot: MvccSnapshot[applied={T|T < 7333066290825621504}]
I20260925 01:23:07.421618 8725 tablet_replica.cc:439] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f60810004e943298bfbe88a0589ee3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 } } peers { permanent_uuid: "b0d200ace75a4d4e9d931a6989ad142b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 32825 } } peers { permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 } } }
I20260925 01:23:07.421967 8725 tablet_replica.cc:442] T 53d6054f30dd49a1b835f42eb5917275 P b0d200ace75a4d4e9d931a6989ad142b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:07.432952 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.427322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.6.236.130:32825)
0925 01:23:07.427465 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.427469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.427486 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:07.427664 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.427668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.427740 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.427869 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:07.428256 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.428980 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.430853 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.431231 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.431316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.431844 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.432378 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.432478 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.432623 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.432690 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:23:07.448993 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.443052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:44585 (local address 127.6.236.130:36445)
0925 01:23:07.443632 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.443651 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.443787 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.444149 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.444152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.444161 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:07.444431 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.444440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.445787 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.445790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.446875 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.446887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.447867 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.447871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.447994 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.448058 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.448508 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.448515 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.448591 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.448684 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
I20260925 01:23:07.449050 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.442971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:36445 (local address 127.6.236.129:44585)
0925 01:23:07.443118 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.443123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.443738 (+ 615us) server_negotiation.cc:407] Connection header received
0925 01:23:07.443951 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.443958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.444049 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.444147 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:07.444594 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.445672 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.447016 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.447695 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.447842 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.448236 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.448417 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.448534 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.448629 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.448737 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 01:23:07.458341 8654 raft_consensus.cc:1261] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Refusing update from remote peer b0d200ace75a4d4e9d931a6989ad142b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.458807 8371 raft_consensus.cc:1261] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Refusing update from remote peer b0d200ace75a4d4e9d931a6989ad142b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:07.459076 8749 consensus_queue.cc:1035] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [LEADER]: Connected to new peer: Peer: permanent_uuid: "225a5bba1cea47f0bb41970cb9259f40" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.459481 8725 consensus_queue.cc:1035] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f60810004e943298bfbe88a0589ee3f" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:07.494632 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.488308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:34935 (local address 127.6.236.130:32825)
0925 01:23:07.488479 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.488485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.489058 (+ 573us) server_negotiation.cc:407] Connection header received
0925 01:23:07.489238 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.489243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.489319 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.489399 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:07.490046 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.490878 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.492917 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.493318 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.493417 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.493921 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.494045 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.494148 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.494294 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.494381 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:23:07.494801 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.488339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:32825 (local address 127.6.236.129:34935)
0925 01:23:07.488968 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:07.488985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:07.489107 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:07.489557 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:07.489566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:07.489579 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:07.489880 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.489890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.491004 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.491009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.492771 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:07.492787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.493484 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.493490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:07.493664 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.493732 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:07.494156 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:07.494173 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 01:23:07.494273 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:23:07.494421 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":320,"threads_started":1}
I20260925 01:23:07.531802 8212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:07.532068 8212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:07.534895 8212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f08ec27cab1349ab8674d0fa9d5e80b6: Sending DeleteTablet for 3 replicas of tablet 8ca3e0e3305f459fbb109e4ade373170
I20260925 01:23:07.535667 8490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:59484
I20260925 01:23:07.535688 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:40278
I20260925 01:23:07.535913 8743 tablet_replica.cc:330] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: stopping tablet replica
I20260925 01:23:07.535951 8742 tablet_replica.cc:330] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: stopping tablet replica
I20260925 01:23:07.536113 8743 raft_consensus.cc:2229] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:07.536151 8742 raft_consensus.cc:2229] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:07.536339 8742 raft_consensus.cc:2258] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.536595 8743 raft_consensus.cc:2258] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.537285 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca3e0e3305f459fbb109e4ade373170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:07 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:46198
I20260925 01:23:07.537549 8741 tablet_replica.cc:330] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: stopping tablet replica
I20260925 01:23:07.537770 8741 raft_consensus.cc:2229] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:07.537814 8743 ts_tablet_manager.cc:1905] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.537950 8741 raft_consensus.cc:2258] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:07.539175 8742 ts_tablet_manager.cc:1905] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.539376 8741 ts_tablet_manager.cc:1905] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:07.544966 8743 ts_tablet_manager.cc:1918] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:07.545128 8743 log.cc:1196] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/8ca3e0e3305f459fbb109e4ade373170
I20260925 01:23:07.545595 8743 ts_tablet_manager.cc:1939] T 8ca3e0e3305f459fbb109e4ade373170 P b0d200ace75a4d4e9d931a6989ad142b: Deleting consensus metadata
I20260925 01:23:07.545893 8742 ts_tablet_manager.cc:1918] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:07.546051 8742 log.cc:1196] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/8ca3e0e3305f459fbb109e4ade373170
I20260925 01:23:07.546448 8742 ts_tablet_manager.cc:1939] T 8ca3e0e3305f459fbb109e4ade373170 P 5f60810004e943298bfbe88a0589ee3f: Deleting consensus metadata
I20260925 01:23:07.547485 8200 catalog_manager.cc:4841] TS 5f60810004e943298bfbe88a0589ee3f (127.6.236.129:44585): tablet 8ca3e0e3305f459fbb109e4ade373170 (table test-table [id=56dcd916270e40beb1441db8cd4a94f8]) successfully deleted
I20260925 01:23:07.549412 8741 ts_tablet_manager.cc:1918] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:07.549628 8741 log.cc:1196] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/8ca3e0e3305f459fbb109e4ade373170
I20260925 01:23:07.550117 8741 ts_tablet_manager.cc:1939] T 8ca3e0e3305f459fbb109e4ade373170 P 225a5bba1cea47f0bb41970cb9259f40: Deleting consensus metadata
I20260925 01:23:07.551250 8197 catalog_manager.cc:4841] TS 225a5bba1cea47f0bb41970cb9259f40 (127.6.236.131:44281): tablet 8ca3e0e3305f459fbb109e4ade373170 (table test-table [id=56dcd916270e40beb1441db8cd4a94f8]) successfully deleted
I20260925 01:23:07.555321 8200 catalog_manager.cc:4841] TS b0d200ace75a4d4e9d931a6989ad142b (127.6.236.130:32825): tablet 8ca3e0e3305f459fbb109e4ade373170 (table test-table [id=56dcd916270e40beb1441db8cd4a94f8]) successfully deleted
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:07.566428 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.548830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.6.236.129:44585)
0925 01:23:07.549004 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.549011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.552116 (+ 3105us) server_negotiation.cc:407] Connection header received
0925 01:23:07.552296 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.552302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.552371 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.552448 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.554050 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.554979 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.556399 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.556843 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.556952 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.560736 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.560748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.560763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.560795 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.563698 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.564353 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.564360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.564366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.564506 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.564876 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.564881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.564884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.565256 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.565546 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.565848 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.566058 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 01:23:07.567109 8416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40300
I20260925 01:23:07.582445 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.570619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.6.236.190:41495)
0925 01:23:07.570802 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.570808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.571367 (+ 559us) server_negotiation.cc:407] Connection header received
0925 01:23:07.571544 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.571550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.571633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.571714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.573031 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.573738 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.574979 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.575393 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.575502 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.577539 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.577544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.577547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.577588 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.579999 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.580591 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.580596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.580599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.580725 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.581061 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.581064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.581066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.581315 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.581625 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.581882 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.582096 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 01:23:07.583180 8212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55872
I20260925 01:23:07.585119 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8285
I20260925 01:23:07.599074 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.587079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.6.236.129:44585)
0925 01:23:07.587405 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.587410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.587427 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:07.587475 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.587479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.587535 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.587620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.589032 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.589800 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.591039 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.591568 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.591690 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.593645 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.593650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.593653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.593684 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.596479 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.597061 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.597067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.597071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.597209 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.597560 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.597565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.597569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.597838 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.598129 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.598450 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.598746 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
W20260925 01:23:07.599614 8416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40312
W20260925 01:23:07.600124 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8285: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:07.600284 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8285
I20260925 01:23:07.620157 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8425
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:07.635206 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.620887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.6.236.130:32825)
0925 01:23:07.621121 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.621127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.621146 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:07.621328 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.621334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.621426 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.621517 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.622950 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.623798 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.625019 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.625517 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.625635 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.629486 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.629503 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.629520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.629555 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.632503 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.633142 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.633149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.633152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.633308 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.633671 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.633675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.633677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.634046 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.634436 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.634641 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.634861 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:23:07.635874 8555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59492
W20260925 01:23:07.636565 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:07.636725 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8425
I20260925 01:23:07.652192 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8564
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:07.665760 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:07.652833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.6.236.131:44281)
0925 01:23:07.653041 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:07.653046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:07.653064 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:23:07.653171 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:07.653175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:07.653248 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:07.653328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:07.654445 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.655154 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.656261 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:07.656674 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:07.656758 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:07.660524 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:07.660539 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:07.660556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:07.660586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:07.663347 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.663890 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.663896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.663899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.664107 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:07.664448 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:07.664453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:07.664455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:07.664855 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:07.665124 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:07.665309 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:23:07.665481 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 01:23:07.666465 8694 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46212
W20260925 01:23:07.667039 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:07.667165 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8564
I20260925 01:23:07.681048 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8181
I20260925 01:23:07.682080 8253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:07.797073 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8181
2026-09-25T01:23:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4171 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8784](info): set up 2 sockets
Sep 25 01:23:07 dist-test-slave-s0b2 krb5kdc[8784](info): commencing operation
krb5kdc: starting...
W20260925 01:23:08.791064 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:08 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:08.951573 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:39375
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:39375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:09.263016 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:09.263289 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:09.263423 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:09.270607 8797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:09.270704 8797 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:23:09.270776 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:09.270851 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:09.270897 8797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:09.270990 8797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:09.281301 8797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:39375
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:09.282583 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:09.283882 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:09.295262 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:09.295321 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:09.295095 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:09.296409 8797 server_base.cc:1034] running on GCE node
I20260925 01:23:09.297168 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:09.298571 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:09.299795 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790299389299723 us; error 45 us; skew 500 ppm
I20260925 01:23:09.300103 8797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:09 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:09.303105 8797 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:09.304454 8797 webserver.cc:460] Webserver started at http://127.6.236.190:43891/ using document root <none> and password file <none>
I20260925 01:23:09.305147 8797 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:09.305248 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:09.305548 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:09.308514 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "c118c7a23a514196b78d92c2257742e3"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "83cd6c6a4d21c1e62bcb983a8ae1d4a7"
server_key_iv: "c65fbc52f4487fb5e9dad25c34baaa7e"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.309307 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "c118c7a23a514196b78d92c2257742e3"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "83cd6c6a4d21c1e62bcb983a8ae1d4a7"
server_key_iv: "c65fbc52f4487fb5e9dad25c34baaa7e"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.314402 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:23:09.317655 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:09.318953 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:09.319118 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "c118c7a23a514196b78d92c2257742e3"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "83cd6c6a4d21c1e62bcb983a8ae1d4a7"
server_key_iv: "c65fbc52f4487fb5e9dad25c34baaa7e"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.319309 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:09.346453 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:09.350339 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:09.350580 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:09.370355 8797 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:39375
I20260925 01:23:09.370376 8873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:39375 every 8 connection(s)
I20260925 01:23:09.371675 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:09.375429 8874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:09.376824 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8797
I20260925 01:23:09.377039 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:09.377858 7090 external_mini_cluster.cc:1428] Setting key a9e74640670bebcc01e1b210a0cbfe8d
I20260925 01:23:09.388046 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Bootstrap starting.
Sep 25 01:23:09 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:09.391260 8874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:09.392248 8874 log.cc:824] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:09.395514 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: No bootstrap required, opened a new log
I20260925 01:23:09.399699 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:09.380005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.6.236.190:39375)
0925 01:23:09.380737 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:09.380749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:09.380796 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:23:09.382401 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:09.382427 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:09.382840 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:09.383197 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:09.384557 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.385864 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.387092 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.387763 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.387928 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:09.391862 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:09.391870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:09.391885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:09.391929 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:09.395389 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.396058 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.396066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.396073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.396284 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.396637 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.396642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.396649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.397317 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:09.397629 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:09.397900 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 01:23:09.398370 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":305,"threads_started":1}
I20260925 01:23:09.404471 8874 raft_consensus.cc:348] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } }
I20260925 01:23:09.404790 8874 raft_consensus.cc:374] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:09.404860 8874 raft_consensus.cc:729] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c118c7a23a514196b78d92c2257742e3, State: Initialized, Role: FOLLOWER
I20260925 01:23:09.405347 8874 consensus_queue.cc:260] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } }
I20260925 01:23:09.405548 8874 raft_consensus.cc:388] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:09.405642 8874 raft_consensus.cc:482] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:09.405761 8874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:09.407779 8874 raft_consensus.cc:504] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } }
I20260925 01:23:09.408313 8874 leader_election.cc:302] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c118c7a23a514196b78d92c2257742e3; no voters:
I20260925 01:23:09.408895 8874 leader_election.cc:288] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:09.409016 8879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:09.409621 8879 raft_consensus.cc:686] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [term 1 LEADER]: Becoming Leader. State: Replica: c118c7a23a514196b78d92c2257742e3, State: Running, Role: LEADER
I20260925 01:23:09.410166 8879 consensus_queue.cc:237] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } }
I20260925 01:23:09.410828 8874 sys_catalog.cc:564] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:09.413317 8880 sys_catalog.cc:455] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c118c7a23a514196b78d92c2257742e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } } }
I20260925 01:23:09.412791 8881 sys_catalog.cc:455] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c118c7a23a514196b78d92c2257742e3. Latest consensus state: current_term: 1 leader_uuid: "c118c7a23a514196b78d92c2257742e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118c7a23a514196b78d92c2257742e3" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39375 } } }
I20260925 01:23:09.414130 8880 sys_catalog.cc:458] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:09.414180 8881 sys_catalog.cc:458] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:09.415953 8888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:09.421885 8888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:09.431133 8888 catalog_manager.cc:1347] Generated new cluster ID: 723f7034aa5e40faa2dbb15e574089d9
I20260925 01:23:09.431245 8888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:09.440497 8888 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:09.442330 8888 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:09.455689 8888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Generated new TSK 0
I20260925 01:23:09.456669 8888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:09.503615 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:39375
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:09.816277 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:09.816565 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:09.816712 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:09.823962 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:09.824079 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:09.824203 8900 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:23:09.824270 8900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:09.824373 8900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:09.833810 8900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:09.835120 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:09.836468 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:09.848927 8905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:09.848991 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:09.848970 8906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:09.849700 8900 server_base.cc:1034] running on GCE node
I20260925 01:23:09.850312 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:09.851204 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:09.852409 8900 hybrid_clock.cc:648] HybridClock initialized: now 1790299389852361 us; error 45 us; skew 500 ppm
I20260925 01:23:09.852707 8900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:09 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:09.855774 8900 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:09.857131 8900 webserver.cc:460] Webserver started at http://127.6.236.129:33743/ using document root <none> and password file <none>
I20260925 01:23:09.857769 8900 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:09.857882 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:09.858165 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:09.860890 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "71c56c57bff34d1ebd6c53b509fda955"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "17cd50273b46904e7ead19a4820dfadc"
server_key_iv: "04a02f72f8e3778820c78380dd26e5d8"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.861619 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "71c56c57bff34d1ebd6c53b509fda955"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "17cd50273b46904e7ead19a4820dfadc"
server_key_iv: "04a02f72f8e3778820c78380dd26e5d8"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.866396 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 01:23:09.869656 8915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:09.870925 8900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:09.871078 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "71c56c57bff34d1ebd6c53b509fda955"
format_stamp: "Formatted at 2026-09-25 01:23:09 on dist-test-slave-s0b2"
server_key: "17cd50273b46904e7ead19a4820dfadc"
server_key_iv: "04a02f72f8e3778820c78380dd26e5d8"
server_key_version: "encryptionkey@0"
I20260925 01:23:09.871263 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:09.882651 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:09.886129 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:09.886397 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:09.888756 8900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:09.888854 8900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:09.888964 8900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:09.889041 8900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:09 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:09.909763 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:09.889382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:44551 (local address 127.6.236.190:39375)
0925 01:23:09.889610 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:09.889618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:09.890877 (+ 1259us) server_negotiation.cc:407] Connection header received
0925 01:23:09.892872 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:09.892877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:09.892966 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:09.893049 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:09.895291 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.896400 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.898881 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.899445 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.899577 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:09.904021 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:09.904029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:09.904034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:09.904075 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:09.906905 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.907568 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.907574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.907578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.907703 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.908116 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.908121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.908124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.908384 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:09.908516 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:09.909190 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 01:23:09.909440 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:23:09.911059 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:09.889856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.129:44551)
0925 01:23:09.890710 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:09.890761 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:09.892602 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:09.893303 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:09.893311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:09.894256 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:09.895081 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:09.895113 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.896577 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.896586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:09.898741 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:09.898757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.899862 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.899874 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:09.900171 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:09.900858 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:09.900892 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:09.903778 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:09.907041 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:09.907049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:09.907064 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:09.907449 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:09.907803 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:09.907808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:09.907812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:09.908004 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:09.908506 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:09.908512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:09.908850 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 01:23:09.909070 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 01:23:09.909548 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260925 01:23:09.930092 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:09.916420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.129:33241)
0925 01:23:09.916981 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:09.916994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:09.917118 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:09.917390 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:09.917394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:09.917818 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:09.918106 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:09.918115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.919432 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.919436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:09.920993 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:09.921012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.921911 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.921917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:09.922143 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:09.922791 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:09.922816 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:09.924041 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:09.926975 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:09.926982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:09.926986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:09.927631 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:09.928104 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:09.928110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:09.928113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:09.928259 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:09.928904 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:09.928907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:09.929140 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 01:23:09.929306 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 01:23:09.929610 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
I20260925 01:23:09.930807 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:09.916354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:33241 (local address 127.6.236.190:39375)
0925 01:23:09.916502 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:09.916506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:09.917080 (+ 574us) server_negotiation.cc:407] Connection header received
0925 01:23:09.917216 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:09.917221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:09.917283 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:09.917356 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:09.918272 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.919259 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.921171 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:09.921722 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:09.921842 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:09.924254 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:09.924261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:09.924266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:09.924305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:09.926831 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.927801 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.927807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.927810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.927960 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:09.928367 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:09.928372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:09.928376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:09.928720 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:09.928853 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:09.930424 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0925 01:23:09.930600 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:23:09.936962 8900 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:35789
I20260925 01:23:09.937008 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:35789 every 8 connection(s)
I20260925 01:23:09.938079 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:09.939888 9036 heartbeater.cc:347] Connected to a master server at 127.6.236.190:39375
I20260925 01:23:09.940073 9036 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:09.940774 9036 heartbeater.cc:511] Master 127.6.236.190:39375 requested a full tablet report, sending...
I20260925 01:23:09.940864 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 8900
I20260925 01:23:09.941075 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:09.941936 7090 external_mini_cluster.cc:1428] Setting key 3de77a0d116cba645487338ea827d0f6
I20260925 01:23:09.943001 8829 ts_manager.cc:194] Registered new tserver with Master: 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789)
I20260925 01:23:09.944442 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:44551
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:09.976742 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:39375
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:10.279999 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:10.280257 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:10.280368 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:10.287314 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:10.287415 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:10.287495 9040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:23:10.287545 9040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:10.287619 9040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:10.296868 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:10.298036 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:10.299402 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:10.311822 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:10.312090 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:10.311790 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:10.312745 9040 server_base.cc:1034] running on GCE node
I20260925 01:23:10.313321 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:10.314280 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:10.315471 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790299390315430 us; error 35 us; skew 500 ppm
I20260925 01:23:10.315742 9040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:10.318567 9040 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:10.319965 9040 webserver.cc:460] Webserver started at http://127.6.236.130:44431/ using document root <none> and password file <none>
I20260925 01:23:10.320604 9040 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:10.320719 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:10.321007 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:10.323941 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "446d9495119a4989b7da34a8b71a2c6d"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "7e706c2ade86798ee1b95e7d51d5f339"
server_key_iv: "7261894000907c6b2be9afd63be0639f"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.324652 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "446d9495119a4989b7da34a8b71a2c6d"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "7e706c2ade86798ee1b95e7d51d5f339"
server_key_iv: "7261894000907c6b2be9afd63be0639f"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.329546 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 01:23:10.332878 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:10.334127 9040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:10.334302 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "446d9495119a4989b7da34a8b71a2c6d"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "7e706c2ade86798ee1b95e7d51d5f339"
server_key_iv: "7261894000907c6b2be9afd63be0639f"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.334486 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:10.352526 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:10.356501 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:10.356753 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:10.359217 9040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:10.359305 9040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:10.359416 9040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:10.359488 9040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:10.383242 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.359763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:53055 (local address 127.6.236.190:39375)
0925 01:23:10.359951 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.359957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.361374 (+ 1417us) server_negotiation.cc:407] Connection header received
0925 01:23:10.363527 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.363533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.363628 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.363754 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:10.366193 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.367196 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.370045 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.370867 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.371001 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.375433 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:10.375441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:10.375446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:10.375487 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:10.378900 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.379818 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.379823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.379827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.379959 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.380473 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.380478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.380482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.380949 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:10.382477 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.382620 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.382878 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:23:10.384189 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.360344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.130:53055)
0925 01:23:10.361204 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.361247 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.363207 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.364207 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.364223 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.365048 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:10.365990 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.366022 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.367389 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.367399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.369847 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.369864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.371108 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.371116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.371380 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.372119 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:10.372146 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:10.375194 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:10.379042 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.379052 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.379070 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.379707 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.380091 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.380100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.380104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.380335 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.381105 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:10.381116 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:10.381616 (+ 500us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.381907 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.382461 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":377,"threads_started":1}
I20260925 01:23:10.405889 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.391591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:54621 (local address 127.6.236.190:39375)
0925 01:23:10.391771 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.391776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.392473 (+ 697us) server_negotiation.cc:407] Connection header received
0925 01:23:10.392699 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.392705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.392773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.392853 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:10.394152 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.395186 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.397127 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.397668 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.397777 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.399947 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:10.399953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:10.399958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:10.399997 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:10.402905 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.403704 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.403711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.403714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.403844 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.404279 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.404284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.404288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.404651 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:10.404804 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.405301 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.405544 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:23:10.406113 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.391726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.130:54621)
0925 01:23:10.392351 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.392375 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.392532 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.392898 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.392903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.393526 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:10.394008 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.394028 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.395365 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.395370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.396988 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.397008 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.397853 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.397859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.398017 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.398705 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:10.398724 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:10.399774 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:10.403023 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.403029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.403033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.403593 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.403958 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.403963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.403967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.404151 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.404782 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:10.404788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:10.405015 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.405201 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.405580 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":340,"threads_started":1}
I20260925 01:23:10.408354 9040 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:42609
I20260925 01:23:10.408403 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:42609 every 8 connection(s)
I20260925 01:23:10.409823 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:10.411573 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9040
I20260925 01:23:10.411744 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:10.411878 9175 heartbeater.cc:347] Connected to a master server at 127.6.236.190:39375
I20260925 01:23:10.412092 9175 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:10.412310 7090 external_mini_cluster.cc:1428] Setting key 545a4600f4ac53a4cb9374577bffd913
I20260925 01:23:10.412761 9175 heartbeater.cc:511] Master 127.6.236.190:39375 requested a full tablet report, sending...
I20260925 01:23:10.414137 8829 ts_manager.cc:194] Registered new tserver with Master: 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609)
I20260925 01:23:10.415020 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:53055
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:10.449355 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:39375
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:45431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:10.756242 9179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:10.756501 9179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:10.756629 9179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:10.763561 9179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:10.763653 9179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:10.763767 9179 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:23:10.763826 9179 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:10.763921 9179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:10.773138 9179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:45431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:39375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:10.774384 9179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:10.775702 9179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:10.787838 9184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:10.787922 9187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:10.787922 9185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:10.788571 9179 server_base.cc:1034] running on GCE node
I20260925 01:23:10.789251 9179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:10.790416 9179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:10.791658 9179 hybrid_clock.cc:648] HybridClock initialized: now 1790299390791586 us; error 41 us; skew 500 ppm
I20260925 01:23:10.792043 9179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:10.795055 9179 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:10.796489 9179 webserver.cc:460] Webserver started at http://127.6.236.131:34113/ using document root <none> and password file <none>
I20260925 01:23:10.797137 9179 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:10.797250 9179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:10.797593 9179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:10.800781 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "533a68f6e73f47dfbe860962347fa3bf"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "51211fb57916807bdd44c4953fff3983"
server_key_iv: "498eb3cce61215ce1dcd3b2b86cb84ea"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.801549 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "533a68f6e73f47dfbe860962347fa3bf"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "51211fb57916807bdd44c4953fff3983"
server_key_iv: "498eb3cce61215ce1dcd3b2b86cb84ea"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.806529 9179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 01:23:10.809788 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:10.811255 9179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:10.811378 9179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "533a68f6e73f47dfbe860962347fa3bf"
format_stamp: "Formatted at 2026-09-25 01:23:10 on dist-test-slave-s0b2"
server_key: "51211fb57916807bdd44c4953fff3983"
server_key_iv: "498eb3cce61215ce1dcd3b2b86cb84ea"
server_key_version: "encryptionkey@0"
I20260925 01:23:10.811540 9179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:10.824823 9179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:10.828562 9179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:10.828799 9179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:10.831316 9179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:10.831413 9179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:10.831568 9179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:10.831655 9179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:10.852977 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.831692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:46573 (local address 127.6.236.190:39375)
0925 01:23:10.831879 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.831885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.833192 (+ 1307us) server_negotiation.cc:407] Connection header received
0925 01:23:10.835511 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.835519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.835620 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.835747 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:10.837715 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.838886 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.841314 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.841761 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.841874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.846448 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:10.846456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:10.846461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:10.846504 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:10.849600 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.850643 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.850649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.850652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.850783 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.851270 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.851272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.851274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.851524 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:10.851633 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.852557 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.852737 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:23:10.854815 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.832216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.131:46573)
0925 01:23:10.833004 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.833057 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.835235 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.835915 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.835923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.836581 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:10.837433 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.837460 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.839070 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.839075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.841177 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.841189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.841997 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.842006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.842304 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.843018 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:10.843053 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:10.846175 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:10.849752 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.849762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.849781 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.850515 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.850883 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.850889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.850893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.851172 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.851636 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:10.851644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:10.852162 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.852470 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.853036 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":331,"threads_started":1}
I20260925 01:23:10.875125 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.861063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:52311 (local address 127.6.236.190:39375)
0925 01:23:10.861203 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.861207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.861792 (+ 585us) server_negotiation.cc:407] Connection header received
0925 01:23:10.861943 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.861948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.862017 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.862095 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:10.863342 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.864431 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.866218 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.866875 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.867022 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.869222 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:10.869229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:10.869233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:10.869263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:10.872305 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.872944 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.872949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.872952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.873146 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.873599 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.873605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.873608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.873932 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:10.874067 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.874544 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.874750 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:23:10.875187 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.861133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.6.236.131:52311)
0925 01:23:10.861692 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.861707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.861837 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.862140 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.862145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.862701 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:10.863123 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.863137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.864571 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.864577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.866076 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.866090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.867076 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.867087 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.867264 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.867952 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:10.867975 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:10.868995 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:10.872453 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.872456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.872459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.872834 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.873276 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:10.873280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:10.873288 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:10.873447 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:10.874046 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:10.874050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:10.874228 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.874411 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.874727 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":294,"threads_started":1}
I20260925 01:23:10.880249 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44293 every 8 connection(s)
I20260925 01:23:10.880448 9179 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44293
I20260925 01:23:10.881606 9179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:10.883383 9314 heartbeater.cc:347] Connected to a master server at 127.6.236.190:39375
I20260925 01:23:10.883595 9314 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:10.884202 9314 heartbeater.cc:511] Master 127.6.236.190:39375 requested a full tablet report, sending...
I20260925 01:23:10.884282 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9179
I20260925 01:23:10.884438 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:10.884852 7090 external_mini_cluster.cc:1428] Setting key 7b0b359f533caa51f76eeebf15d513a9
I20260925 01:23:10.885527 8829 ts_manager.cc:194] Registered new tserver with Master: 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:10.886526 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:46573
I20260925 01:23:10.887827 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:23:10 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:10.929579 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.917016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.6.236.190:39375)
0925 01:23:10.917181 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.917185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.917533 (+ 348us) server_negotiation.cc:407] Connection header received
0925 01:23:10.917650 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.917655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.917801 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.917921 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:10.919261 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.919989 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.921123 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.921540 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.921636 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.925278 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:10.925283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:10.925286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:10.925321 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:10.927610 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.928153 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.928156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.928158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.928276 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:10.928536 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:10.928539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:10.928541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:10.928793 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:10.928961 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.929180 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.929354 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:23:10.933423 8828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:10.940361 8828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:10.947022 9036 heartbeater.cc:503] Master 127.6.236.190:39375 was elected leader, sending a full tablet report...
I20260925 01:23:10.978787 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.951631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.0.0.1:48394)
0925 01:23:10.963151 (+ 11520us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.963221 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.963416 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.971127 (+ 7711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.971135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.971173 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:10.971620 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.971635 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.973428 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.973433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.976017 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.976033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.977812 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.977818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.978022 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.978094 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.978197 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.978407 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11215,"thread_start_us":12108,"threads_started":1}
I20260925 01:23:10.980237 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.951866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.6.236.130:42609)
0925 01:23:10.970585 (+ 18719us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.970593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.970619 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:23:10.970701 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.970707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.970911 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.971091 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:10.971835 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.973290 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.976226 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.977586 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.979533 (+ 1947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.979750 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.979818 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.979978 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18574,"thread_start_us":18448,"threads_started":1}
I20260925 01:23:10.980808 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.951872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44293 (local address 127.0.0.1:33830)
0925 01:23:10.964024 (+ 12152us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.964040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.964151 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.972739 (+ 8588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.972744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.972769 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:10.973149 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.973160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.975239 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.975244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.977472 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.977489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.979898 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.979905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.980106 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.980144 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.980242 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.980377 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12030,"thread_start_us":10833,"threads_started":1}
I20260925 01:23:10.982271 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.952184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.6.236.131:44293)
0925 01:23:10.971786 (+ 19602us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.971795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.971825 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:23:10.972025 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.972032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.972309 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.972483 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:10.973357 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.975075 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.977655 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.979038 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.979162 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.979394 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.981714 (+ 2320us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.981874 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19419,"thread_start_us":19255,"threads_started":1}
I20260925 01:23:10.985581 9244 tablet_service.cc:1467] Processing CreateTablet for tablet a6bcd3d51a01484baf9d13c305964148 (DEFAULT_TABLE table=test-table [id=9a7a16dbb3854870a80c428b428e8ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:10.985580 9105 tablet_service.cc:1467] Processing CreateTablet for tablet a6bcd3d51a01484baf9d13c305964148 (DEFAULT_TABLE table=test-table [id=9a7a16dbb3854870a80c428b428e8ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:10.986572 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.964360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35789 (local address 127.0.0.1:42462)
0925 01:23:10.979275 (+ 14915us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:10.979294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:10.979384 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:10.980385 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:10.980389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:10.980413 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:10.980569 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.980580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.982427 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.982433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.984061 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:10.984076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.985824 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.985830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:10.986043 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.986078 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:10.986161 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 01:23:10.986325 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14823,"thread_start_us":16873,"threads_started":1}
I20260925 01:23:10.987067 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bcd3d51a01484baf9d13c305964148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:10.987080 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bcd3d51a01484baf9d13c305964148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:10.987401 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:10.964424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.6.236.129:35789)
0925 01:23:10.979795 (+ 15371us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:10.979802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:10.979828 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:23:10.979911 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:10.979919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:10.980177 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:10.980727 (+ 550us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:10.980947 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.982268 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.984237 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:10.985600 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:10.986428 (+ 828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:10.986620 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:10.986844 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:23:10.986999 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15220,"thread_start_us":15088,"threads_started":1}
I20260925 01:23:10.990908 8966 tablet_service.cc:1467] Processing CreateTablet for tablet a6bcd3d51a01484baf9d13c305964148 (DEFAULT_TABLE table=test-table [id=9a7a16dbb3854870a80c428b428e8ab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:10.992271 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6bcd3d51a01484baf9d13c305964148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:10.998910 9333 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Bootstrap starting.
I20260925 01:23:11.000735 9334 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Bootstrap starting.
I20260925 01:23:11.002370 9333 tablet_bootstrap.cc:654] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.003969 9334 tablet_bootstrap.cc:654] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.004319 9333 log.cc:824] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:11.005172 9334 log.cc:824] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:11.007354 9333 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: No bootstrap required, opened a new log
I20260925 01:23:11.007622 9333 ts_tablet_manager.cc:1397] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 01:23:11.008541 9334 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: No bootstrap required, opened a new log
I20260925 01:23:11.008644 9335 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Bootstrap starting.
I20260925 01:23:11.008800 9334 ts_tablet_manager.cc:1397] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 01:23:11.011538 9335 tablet_bootstrap.cc:654] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.012504 9335 log.cc:824] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:11.015259 9335 tablet_bootstrap.cc:492] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: No bootstrap required, opened a new log
I20260925 01:23:11.015484 9335 ts_tablet_manager.cc:1397] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 01:23:11.017807 9333 raft_consensus.cc:348] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.018224 9333 raft_consensus.cc:374] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.018340 9333 raft_consensus.cc:729] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.018913 9333 consensus_queue.cc:260] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.018983 9334 raft_consensus.cc:348] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.019352 9334 raft_consensus.cc:374] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.019451 9334 raft_consensus.cc:729] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.020118 9334 consensus_queue.cc:260] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.021168 9175 heartbeater.cc:503] Master 127.6.236.190:39375 was elected leader, sending a full tablet report...
I20260925 01:23:11.021824 9334 ts_tablet_manager.cc:1428] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260925 01:23:11.024326 9335 raft_consensus.cc:348] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.024734 9335 raft_consensus.cc:374] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.024842 9335 raft_consensus.cc:729] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.025400 9335 consensus_queue.cc:260] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.025840 9314 heartbeater.cc:503] Master 127.6.236.190:39375 was elected leader, sending a full tablet report...
I20260925 01:23:11.026736 9333 ts_tablet_manager.cc:1428] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Time spent starting tablet: real 0.019s user 0.010s sys 0.010s
I20260925 01:23:11.028191 9335 ts_tablet_manager.cc:1428] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 01:23:11.102104 9340 raft_consensus.cc:482] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:11.102449 9340 raft_consensus.cc:504] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.104161 9340 leader_election.cc:288] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293), 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789)
I20260925 01:23:11.112457 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.104577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:48459 (local address 127.6.236.131:44293)
0925 01:23:11.104786 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.104793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.105516 (+ 723us) server_negotiation.cc:407] Connection header received
0925 01:23:11.105744 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.105750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.105829 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.105895 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:11.106627 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.107349 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.110151 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.111405 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.111597 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.111672 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.112059 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.112197 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:23:11.112493 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.104789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:50113 (local address 127.6.236.129:35789)
0925 01:23:11.104998 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.105004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.105643 (+ 639us) server_negotiation.cc:407] Connection header received
0925 01:23:11.105810 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.105814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.105876 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.105966 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:11.106655 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.107443 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.110385 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.111631 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.111843 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.111939 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.112095 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.112218 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:23:11.112939 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.104579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44293 (local address 127.6.236.130:48459)
0925 01:23:11.105390 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.105408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.105553 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.105932 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.105937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.105997 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:11.106453 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.106470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.107484 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.107489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.109999 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.110008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.111640 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.111646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.111830 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.111865 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.112019 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.112192 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":145,"thread_start_us":416,"threads_started":1}
I20260925 01:23:11.112960 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.104615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35789 (local address 127.6.236.130:50113)
0925 01:23:11.105532 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.105550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.105680 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.105994 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.105998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.106023 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:11.106484 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.106492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.107580 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.107584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.109989 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.110005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.111817 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.111823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.111964 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.111993 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.112068 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.112196 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":112,"thread_start_us":587,"threads_started":1}
I20260925 01:23:11.114030 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bcd3d51a01484baf9d13c305964148" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf" is_pre_election: true
I20260925 01:23:11.114018 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bcd3d51a01484baf9d13c305964148" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955" is_pre_election: true
I20260925 01:23:11.114549 9264 raft_consensus.cc:2454] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 0.
I20260925 01:23:11.114558 8991 raft_consensus.cc:2454] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 0.
I20260925 01:23:11.115170 9058 leader_election.cc:302] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf; no voters:
I20260925 01:23:11.115569 9340 raft_consensus.cc:2781] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:11.115711 9340 raft_consensus.cc:482] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:11.115808 9340 raft_consensus.cc:3037] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.117960 9340 raft_consensus.cc:504] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.118562 9340 leader_election.cc:288] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 election: Requested vote from peers 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293), 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789)
I20260925 01:23:11.118956 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bcd3d51a01484baf9d13c305964148" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf"
I20260925 01:23:11.118961 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bcd3d51a01484baf9d13c305964148" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955"
I20260925 01:23:11.119133 8991 raft_consensus.cc:3037] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.119119 9264 raft_consensus.cc:3037] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.121214 8991 raft_consensus.cc:2454] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 1.
I20260925 01:23:11.121304 9264 raft_consensus.cc:2454] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 1.
I20260925 01:23:11.121645 9056 leader_election.cc:302] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:11.121934 9340 raft_consensus.cc:2781] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:11.122372 9340 raft_consensus.cc:686] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 1 LEADER]: Becoming Leader. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Running, Role: LEADER
I20260925 01:23:11.122867 9340 consensus_queue.cc:237] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } }
I20260925 01:23:11.127095 8829 catalog_manager.cc:5495] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130). New cstate: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } health_report { overall_health: UNKNOWN } } }
W20260925 01:23:11.133070 9315 tablet.cc:2369] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:23:11.161851 9176 tablet.cc:2369] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 01:23:11 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299389, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:11.182536 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.164637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:39375 (local address 127.0.0.1:34782)
0925 01:23:11.165227 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.165243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.165368 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.165635 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.165638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.166350 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:11.166635 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.166645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.167605 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.167609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.168593 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.168604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.169322 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.169328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.169460 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.175501 (+ 6041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:11.175525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:11.177715 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:11.180369 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:11.180380 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:11.180385 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:11.180778 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:11.181081 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:11.181084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:11.181086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:11.181215 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:11.181671 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:11.181678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:11.181835 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.181965 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.182157 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260925 01:23:11.182580 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.164551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.6.236.190:39375)
0925 01:23:11.164763 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.164769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.165323 (+ 554us) server_negotiation.cc:407] Connection header received
0925 01:23:11.165457 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.165461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.165522 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.165605 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:11.166765 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.167482 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.168705 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.169126 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.169212 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.177894 (+ 8682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:11.177899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:11.177902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:11.177932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:11.180247 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:11.180869 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:11.180872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:11.180874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:11.180997 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:11.181295 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:11.181298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:11.181300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:11.181561 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:11.181664 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.182049 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.182298 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":76}
I20260925 01:23:11.187443 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34782:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:23:11.188732 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 01:23:11.190500 9037 tablet.cc:2369] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:11.201638 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 67464c79100b4c42ac8fe6096baaee85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=153ba2785da44208926249696551850c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:11.201628 8966 tablet_service.cc:1467] Processing CreateTablet for tablet 67464c79100b4c42ac8fe6096baaee85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=153ba2785da44208926249696551850c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:11.202653 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67464c79100b4c42ac8fe6096baaee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.202665 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67464c79100b4c42ac8fe6096baaee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.203140 9244 tablet_service.cc:1467] Processing CreateTablet for tablet 67464c79100b4c42ac8fe6096baaee85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=153ba2785da44208926249696551850c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:11.204001 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67464c79100b4c42ac8fe6096baaee85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.208529 9334 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: Bootstrap starting.
I20260925 01:23:11.209877 9335 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: Bootstrap starting.
I20260925 01:23:11.209925 9334 tablet_bootstrap.cc:654] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.210744 9333 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: Bootstrap starting.
I20260925 01:23:11.211653 9335 tablet_bootstrap.cc:654] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.212244 9333 tablet_bootstrap.cc:654] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.214407 9333 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: No bootstrap required, opened a new log
I20260925 01:23:11.214571 9333 ts_tablet_manager.cc:1397] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:23:11.215344 9333 raft_consensus.cc:348] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.215535 9333 raft_consensus.cc:374] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.215618 9333 raft_consensus.cc:729] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.215834 9333 consensus_queue.cc:260] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.216385 9333 ts_tablet_manager.cc:1428] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:23:11.216328 9339 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.216737 9339 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.218228 9335 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: No bootstrap required, opened a new log
I20260925 01:23:11.218408 9334 tablet_bootstrap.cc:492] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: No bootstrap required, opened a new log
I20260925 01:23:11.218405 9335 ts_tablet_manager.cc:1397] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260925 01:23:11.218539 9334 ts_tablet_manager.cc:1397] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 01:23:11.219118 9335 raft_consensus.cc:348] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.219349 9335 raft_consensus.cc:374] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.219254 9334 raft_consensus.cc:348] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.219427 9335 raft_consensus.cc:729] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.219476 9334 raft_consensus.cc:374] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.219552 9334 raft_consensus.cc:729] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.219637 9335 consensus_queue.cc:260] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.220216 9335 ts_tablet_manager.cc:1428] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:23:11.220088 9341 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.220479 9341 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.219784 9334 consensus_queue.cc:260] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.221448 9334 ts_tablet_manager.cc:1428] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:11.221293 9340 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.221606 9340 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.254848 9340 raft_consensus.cc:482] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:11.255038 9340 raft_consensus.cc:504] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.255719 9340 leader_election.cc:288] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:11.256217 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67464c79100b4c42ac8fe6096baaee85" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955" is_pre_election: true
I20260925 01:23:11.256315 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67464c79100b4c42ac8fe6096baaee85" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf" is_pre_election: true
I20260925 01:23:11.256472 8991 raft_consensus.cc:2454] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 0.
I20260925 01:23:11.256595 9264 raft_consensus.cc:2454] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 0.
I20260925 01:23:11.256910 9056 leader_election.cc:302] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:11.257217 9340 raft_consensus.cc:2781] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:11.257315 9340 raft_consensus.cc:482] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:11.257411 9340 raft_consensus.cc:3037] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.259416 9340 raft_consensus.cc:504] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.259944 9340 leader_election.cc:288] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 election: Requested vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:11.260322 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67464c79100b4c42ac8fe6096baaee85" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955"
I20260925 01:23:11.260363 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67464c79100b4c42ac8fe6096baaee85" candidate_uuid: "446d9495119a4989b7da34a8b71a2c6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf"
I20260925 01:23:11.260512 8991 raft_consensus.cc:3037] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.260521 9264 raft_consensus.cc:3037] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.262449 9264 raft_consensus.cc:2454] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 1.
I20260925 01:23:11.262580 8991 raft_consensus.cc:2454] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446d9495119a4989b7da34a8b71a2c6d in term 1.
I20260925 01:23:11.262837 9058 leader_election.cc:302] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf; no voters:
I20260925 01:23:11.263072 9340 raft_consensus.cc:2781] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:11.263242 9340 raft_consensus.cc:686] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [term 1 LEADER]: Becoming Leader. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Running, Role: LEADER
I20260925 01:23:11.263465 9340 consensus_queue.cc:237] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.265142 9344 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: TxnStatusTablet state changed. Reason: New leader 446d9495119a4989b7da34a8b71a2c6d. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.265522 9344 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:11.266144 9356 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:23:11.266423 9356 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:23:11.266390 8829 catalog_manager.cc:5495] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130). New cstate: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:11.293530 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.287416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.6.236.130:42609)
0925 01:23:11.287615 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.287622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.287798 (+ 176us) server_negotiation.cc:407] Connection header received
0925 01:23:11.288055 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.288059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.288129 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.288241 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.288622 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.289345 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.291301 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.291759 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.291880 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.292388 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.292908 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.293020 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.293139 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.293216 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:23:11.293612 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.287432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.0.0.1:48408)
0925 01:23:11.287711 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.287732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.287880 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.288289 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.288295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.288309 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:11.288496 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.288508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.289486 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.289492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.291122 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.291138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.291908 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.291915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.292116 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.292188 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.293013 (+ 825us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.293034 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.293124 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.293289 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:23:11.301201 8991 raft_consensus.cc:1261] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Refusing update from remote peer 446d9495119a4989b7da34a8b71a2c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.301201 9264 raft_consensus.cc:1261] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Refusing update from remote peer 446d9495119a4989b7da34a8b71a2c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.302281 9344 consensus_queue.cc:1035] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.302580 9340 consensus_queue.cc:1035] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.313758 9341 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: TxnStatusTablet state changed. Reason: New leader 446d9495119a4989b7da34a8b71a2c6d. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.314075 9341 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.316897 9359 mvcc.cc:205] Tried to move back new op lower bound from 7333066306759856128 to 7333066306617524224. Current Snapshot: MvccSnapshot[applied={T|T < 7333066306759856128}]
I20260925 01:23:11.317410 9340 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.317718 9340 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:11.321331 9339 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: TxnStatusTablet state changed. Reason: New leader 446d9495119a4989b7da34a8b71a2c6d. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.321641 9339 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.323931 9341 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.324170 9341 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 71c56c57bff34d1ebd6c53b509fda955: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.327363 9344 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.327634 9344 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 446d9495119a4989b7da34a8b71a2c6d: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:11.351807 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.344781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.6.236.130:42609)
0925 01:23:11.344972 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.344978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.344995 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:11.345058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.345063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.345145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.345227 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.345789 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.347127 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.349186 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.349837 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.349951 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.350566 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.350742 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.350825 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.351377 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.351456 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:11.364586 9339 tablet_replica.cc:439] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446d9495119a4989b7da34a8b71a2c6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } } }
I20260925 01:23:11.364876 9339 tablet_replica.cc:442] T 67464c79100b4c42ac8fe6096baaee85 P 533a68f6e73f47dfbe860962347fa3bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:11.381346 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.376165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:46459 (local address 127.6.236.130:42609)
0925 01:23:11.376365 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.376371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.376522 (+ 151us) server_negotiation.cc:407] Connection header received
0925 01:23:11.376759 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.376764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.376828 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.376894 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.377409 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.378117 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.379697 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.380157 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.380262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.380693 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.380806 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.380878 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.380994 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.381070 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 01:23:11.381491 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.375780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.6.236.130:46459)
0925 01:23:11.376431 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.376450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.376641 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.376923 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.376927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.376937 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:11.377258 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.377267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.378266 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.378272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.379568 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.379581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.380321 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.380328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.380462 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.380521 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.380891 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.380901 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.380972 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.381097 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":4,"thread_start_us":451,"threads_started":1}
I20260925 01:23:11.391829 8991 raft_consensus.cc:1261] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Refusing update from remote peer 446d9495119a4989b7da34a8b71a2c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.392410 9264 raft_consensus.cc:1261] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Refusing update from remote peer 446d9495119a4989b7da34a8b71a2c6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.392315 9340 consensus_queue.cc:1035] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.393110 9340 consensus_queue.cc:1035] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.401602 9362 mvcc.cc:205] Tried to move back new op lower bound from 7333066307136679936 to 7333066306043068416. Current Snapshot: MvccSnapshot[applied={T|T < 7333066307136679936}]
I20260925 01:23:11.462064 8829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:11.462356 8829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:11.465811 8829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Sending DeleteTablet for 3 replicas of tablet a6bcd3d51a01484baf9d13c305964148
I20260925 01:23:11.466791 9104 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bcd3d51a01484baf9d13c305964148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:11 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:48394
I20260925 01:23:11.466867 8966 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bcd3d51a01484baf9d13c305964148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:11 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42462
I20260925 01:23:11.467689 9375 tablet_replica.cc:330] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: stopping tablet replica
I20260925 01:23:11.467741 9244 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bcd3d51a01484baf9d13c305964148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:11 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:33830
I20260925 01:23:11.468230 9375 raft_consensus.cc:2229] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:11.468641 9375 raft_consensus.cc:2258] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:11.469565 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:11.470192 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:11.470975 9375 ts_tablet_manager.cc:1905] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:11.477530 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d5581cb1449292e79720bf07513f (DEFAULT_TABLE table=test-table [id=777ffcc2b21542ef9c3f44d8dd799153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:11.477975 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b8d5581cb1449292e79720bf07513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.479192 9375 ts_tablet_manager.cc:1918] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:11.479445 9375 log.cc:1196] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/a6bcd3d51a01484baf9d13c305964148
I20260925 01:23:11.480033 9375 ts_tablet_manager.cc:1939] T a6bcd3d51a01484baf9d13c305964148 P 71c56c57bff34d1ebd6c53b509fda955: Deleting consensus metadata
I20260925 01:23:11.480682 9377 tablet_replica.cc:330] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: stopping tablet replica
I20260925 01:23:11.481129 9377 raft_consensus.cc:2229] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:11.481504 9377 raft_consensus.cc:2258] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:11.481902 9243 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d5581cb1449292e79720bf07513f (DEFAULT_TABLE table=test-table [id=777ffcc2b21542ef9c3f44d8dd799153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:11.482301 9376 tablet_replica.cc:330] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: stopping tablet replica
I20260925 01:23:11.482347 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b8d5581cb1449292e79720bf07513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.482401 8966 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d5581cb1449292e79720bf07513f (DEFAULT_TABLE table=test-table [id=777ffcc2b21542ef9c3f44d8dd799153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:11.482689 9376 raft_consensus.cc:2229] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:11.482774 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b8d5581cb1449292e79720bf07513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:11.483302 9376 raft_consensus.cc:2258] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:11.487732 9333 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Bootstrap starting.
I20260925 01:23:11.489233 9333 tablet_bootstrap.cc:654] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.489399 9376 ts_tablet_manager.cc:1905] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:11.494441 8813 catalog_manager.cc:4841] TS 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789): tablet a6bcd3d51a01484baf9d13c305964148 (table test-table [id=9a7a16dbb3854870a80c428b428e8ab6]) successfully deleted
I20260925 01:23:11.495371 9377 ts_tablet_manager.cc:1905] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:11.498461 9335 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Bootstrap starting.
I20260925 01:23:11.499190 9333 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: No bootstrap required, opened a new log
I20260925 01:23:11.499394 9333 ts_tablet_manager.cc:1397] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260925 01:23:11.500078 9335 tablet_bootstrap.cc:654] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.500239 9333 raft_consensus.cc:348] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.500578 9333 raft_consensus.cc:374] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.500694 9333 raft_consensus.cc:729] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.501013 9333 consensus_queue.cc:260] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.501638 9376 ts_tablet_manager.cc:1918] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:11.501791 9376 log.cc:1196] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/a6bcd3d51a01484baf9d13c305964148
I20260925 01:23:11.502622 9333 ts_tablet_manager.cc:1428] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 01:23:11.503620 9334 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Bootstrap starting.
I20260925 01:23:11.503973 9376 ts_tablet_manager.cc:1939] T a6bcd3d51a01484baf9d13c305964148 P 446d9495119a4989b7da34a8b71a2c6d: Deleting consensus metadata
I20260925 01:23:11.505337 9334 tablet_bootstrap.cc:654] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:11.506430 8813 catalog_manager.cc:4841] TS 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609): tablet a6bcd3d51a01484baf9d13c305964148 (table test-table [id=9a7a16dbb3854870a80c428b428e8ab6]) successfully deleted
I20260925 01:23:11.507380 9334 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: No bootstrap required, opened a new log
I20260925 01:23:11.507532 9334 ts_tablet_manager.cc:1397] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 01:23:11.508260 9334 raft_consensus.cc:348] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.508493 9334 raft_consensus.cc:374] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.508589 9334 raft_consensus.cc:729] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.508817 9334 consensus_queue.cc:260] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.509421 9334 ts_tablet_manager.cc:1428] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 01:23:11.513355 9377 ts_tablet_manager.cc:1918] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:11.513513 9377 log.cc:1196] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/a6bcd3d51a01484baf9d13c305964148
I20260925 01:23:11.514003 9377 ts_tablet_manager.cc:1939] T a6bcd3d51a01484baf9d13c305964148 P 533a68f6e73f47dfbe860962347fa3bf: Deleting consensus metadata
I20260925 01:23:11.517462 9335 tablet_bootstrap.cc:492] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: No bootstrap required, opened a new log
I20260925 01:23:11.517620 9335 ts_tablet_manager.cc:1397] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20260925 01:23:11.518085 8815 catalog_manager.cc:4841] TS 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293): tablet a6bcd3d51a01484baf9d13c305964148 (table test-table [id=9a7a16dbb3854870a80c428b428e8ab6]) successfully deleted
I20260925 01:23:11.518306 9335 raft_consensus.cc:348] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.518507 9335 raft_consensus.cc:374] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:11.518585 9335 raft_consensus.cc:729] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Initialized, Role: FOLLOWER
I20260925 01:23:11.518818 9335 consensus_queue.cc:260] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.519351 9335 ts_tablet_manager.cc:1428] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:23:11.809005 9339 raft_consensus.cc:482] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:11.809219 9339 raft_consensus.cc:504] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.810787 9339 leader_election.cc:288] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609)
I20260925 01:23:11.818423 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.811235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:44035 (local address 127.6.236.129:35789)
0925 01:23:11.812383 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.812390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.812418 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:23:11.812498 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.812503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.812575 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.812672 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:11.813321 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.814566 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.816640 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.817453 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.817625 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.817706 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.817963 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.818103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":1027,"threads_started":1}
I20260925 01:23:11.818471 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.811051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35789 (local address 127.6.236.131:44035)
0925 01:23:11.811570 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.811589 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.811719 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.812689 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.812694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.812868 (+ 174us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:11.813130 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.813144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.814706 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.814711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.816499 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.816511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.817627 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.817633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.817801 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.817837 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.817935 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.818068 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":63,"thread_start_us":284,"threads_started":1}
I20260925 01:23:11.819537 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.811736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.6.236.131:42127)
0925 01:23:11.812742 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.812761 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.813031 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.813380 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.813384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.813407 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:11.813745 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.813755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.815032 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.815038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.817214 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.817230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.818796 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.818801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.818971 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.818996 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.819071 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.819199 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":26,"thread_start_us":745,"threads_started":1}
I20260925 01:23:11.819511 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d5581cb1449292e79720bf07513f" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955" is_pre_election: true
I20260925 01:23:11.819798 8990 raft_consensus.cc:2454] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 0.
I20260925 01:23:11.820158 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.812069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:42127 (local address 127.6.236.130:42609)
0925 01:23:11.812245 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.812251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.812847 (+ 596us) server_negotiation.cc:407] Connection header received
0925 01:23:11.813186 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.813191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.813277 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.813357 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:11.813904 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.814844 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.817396 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.818643 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.818787 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.818861 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.819105 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.819226 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:23:11.820294 9195 leader_election.cc:302] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533a68f6e73f47dfbe860962347fa3bf, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:11.820678 9339 raft_consensus.cc:2781] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:11.820788 9339 raft_consensus.cc:482] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:11.820876 9339 raft_consensus.cc:3037] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.821034 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d5581cb1449292e79720bf07513f" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d" is_pre_election: true
I20260925 01:23:11.821502 9130 raft_consensus.cc:2454] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 0.
I20260925 01:23:11.823402 9339 raft_consensus.cc:504] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.824090 9339 leader_election.cc:288] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 election: Requested vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609)
I20260925 01:23:11.824667 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d5581cb1449292e79720bf07513f" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d"
I20260925 01:23:11.824681 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d5581cb1449292e79720bf07513f" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955"
I20260925 01:23:11.824884 8990 raft_consensus.cc:3037] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.824929 9130 raft_consensus.cc:3037] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:11.827965 8990 raft_consensus.cc:2454] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 1.
I20260925 01:23:11.828052 9130 raft_consensus.cc:2454] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 1.
I20260925 01:23:11.828495 9195 leader_election.cc:302] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf; no voters:
I20260925 01:23:11.828775 9339 raft_consensus.cc:2781] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:11.829216 9339 raft_consensus.cc:686] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 1 LEADER]: Becoming Leader. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Running, Role: LEADER
I20260925 01:23:11.829741 9339 consensus_queue.cc:237] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:11.833221 8829 catalog_manager.cc:5495] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131). New cstate: current_term: 1 leader_uuid: "533a68f6e73f47dfbe860962347fa3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } health_report { overall_health: HEALTHY } } }
I20260925 01:23:11.875125 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.868145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.6.236.131:44293)
0925 01:23:11.868709 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.868715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.869011 (+ 296us) server_negotiation.cc:407] Connection header received
0925 01:23:11.869181 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.869186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.869276 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.869438 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.869806 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.870979 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.872327 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.872966 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.873087 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.873432 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.874299 (+ 867us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.874505 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.874655 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.874739 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":312,"threads_started":1}
I20260925 01:23:11.900549 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.893473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36047 (local address 127.6.236.130:42609)
0925 01:23:11.893652 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.893657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.893939 (+ 282us) server_negotiation.cc:407] Connection header received
0925 01:23:11.894217 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.894223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.894341 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.894427 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.895111 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.896284 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.898340 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.899020 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.899121 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.899722 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.899895 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.899969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.900176 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.900262 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:23:11.901553 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.893063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.6.236.131:36047)
0925 01:23:11.893837 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.893857 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.894066 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.894591 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.894596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.894609 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:11.894917 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.894937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.896456 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.896461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.898159 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.898173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.899301 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.899307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.899475 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.899535 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.900056 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.900065 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.900998 (+ 933us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.901150 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":596,"threads_started":1}
I20260925 01:23:11.910689 8990 raft_consensus.cc:1261] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Refusing update from remote peer 533a68f6e73f47dfbe860962347fa3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.910842 9130 raft_consensus.cc:1261] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Refusing update from remote peer 533a68f6e73f47dfbe860962347fa3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:11.911448 9384 consensus_queue.cc:1035] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.911803 9339 consensus_queue.cc:1035] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:11.949113 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.937636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:52711 (local address 127.6.236.131:44293)
0925 01:23:11.937798 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:11.937804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:11.942805 (+ 5001us) server_negotiation.cc:407] Connection header received
0925 01:23:11.943063 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:11.943069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:11.943146 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:11.943235 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:11.943840 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.944934 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.947220 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.947827 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.947954 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.948430 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.948549 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.948622 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:11.948767 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 01:23:11.948849 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:23:11.949378 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:11.936362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44293 (local address 127.6.236.130:52711)
0925 01:23:11.942712 (+ 6350us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:11.942730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:11.942907 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:11.943270 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:11.943275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:11.943288 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:11.943632 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.943645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.945258 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.945263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.947005 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:11.947026 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:11.948020 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:11.948027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:11.948221 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:11.948272 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:11.948648 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:11.948661 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 01:23:11.948753 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 01:23:11.948895 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6199,"thread_start_us":6052,"threads_started":1}
W20260925 01:23:11.950610 9362 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:23:13.557143 8829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:13.557358 8829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:13.561180 8829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Sending DeleteTablet for 3 replicas of tablet 92b8d5581cb1449292e79720bf07513f
I20260925 01:23:13.562007 8966 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d5581cb1449292e79720bf07513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42462
I20260925 01:23:13.562134 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d5581cb1449292e79720bf07513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:48394
I20260925 01:23:13.562172 9243 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d5581cb1449292e79720bf07513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:33830
I20260925 01:23:13.562691 9420 tablet_replica.cc:330] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: stopping tablet replica
I20260925 01:23:13.562893 9420 raft_consensus.cc:2229] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:13.563119 9420 raft_consensus.cc:2258] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.567219 9420 ts_tablet_manager.cc:1905] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:13.574678 9420 ts_tablet_manager.cc:1918] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:23:13.574904 9420 log.cc:1196] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/92b8d5581cb1449292e79720bf07513f
I20260925 01:23:13.575443 9420 ts_tablet_manager.cc:1939] T 92b8d5581cb1449292e79720bf07513f P 71c56c57bff34d1ebd6c53b509fda955: Deleting consensus metadata
I20260925 01:23:13.576828 8813 catalog_manager.cc:4841] TS 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789): tablet 92b8d5581cb1449292e79720bf07513f (table test-table [id=777ffcc2b21542ef9c3f44d8dd799153]) successfully deleted
I20260925 01:23:13.577371 9421 tablet_replica.cc:330] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: stopping tablet replica
I20260925 01:23:13.577565 9421 raft_consensus.cc:2229] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:13.577786 9421 raft_consensus.cc:2258] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.578451 9422 tablet_replica.cc:330] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: stopping tablet replica
I20260925 01:23:13.578665 9422 raft_consensus.cc:2229] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:13.578858 9421 ts_tablet_manager.cc:1905] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:13.579257 9422 raft_consensus.cc:2258] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.580103 9422 ts_tablet_manager.cc:1905] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:13.585309 9421 ts_tablet_manager.cc:1918] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:23:13.585449 9421 log.cc:1196] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/92b8d5581cb1449292e79720bf07513f
I20260925 01:23:13.585937 9421 ts_tablet_manager.cc:1939] T 92b8d5581cb1449292e79720bf07513f P 446d9495119a4989b7da34a8b71a2c6d: Deleting consensus metadata
I20260925 01:23:13.586411 9422 ts_tablet_manager.cc:1918] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:23:13.586560 9422 log.cc:1196] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/92b8d5581cb1449292e79720bf07513f
I20260925 01:23:13.586999 9422 ts_tablet_manager.cc:1939] T 92b8d5581cb1449292e79720bf07513f P 533a68f6e73f47dfbe860962347fa3bf: Deleting consensus metadata
I20260925 01:23:13.587136 8813 catalog_manager.cc:4841] TS 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609): tablet 92b8d5581cb1449292e79720bf07513f (table test-table [id=777ffcc2b21542ef9c3f44d8dd799153]) successfully deleted
I20260925 01:23:13.588024 8815 catalog_manager.cc:4841] TS 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293): tablet 92b8d5581cb1449292e79720bf07513f (table test-table [id=777ffcc2b21542ef9c3f44d8dd799153]) successfully deleted
Sep 25 01:23:13 dist-test-slave-s0b2 krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:23:13 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:13.617435 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.603738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.6.236.190:39375)
0925 01:23:13.604304 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:13.604309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:13.604334 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:23:13.604535 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:13.604540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:13.604607 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:13.604705 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:13.606014 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.606816 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.608032 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.608519 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.608656 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.612693 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:13.612698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:13.612702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:13.612734 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:13.615283 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:13.615863 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:13.615868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:13.615871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:13.616016 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:13.616335 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:13.616337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:13.616339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:13.616625 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:13.616795 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:13.617017 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:23:13.617176 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":283,"threads_started":1}
I20260925 01:23:13.620749 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:13.621337 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:13.627683 8966 tablet_service.cc:1467] Processing CreateTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (DEFAULT_TABLE table=test-table [id=a324a2c55145402188500ec25b8a44e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.627674 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (DEFAULT_TABLE table=test-table [id=a324a2c55145402188500ec25b8a44e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.627948 9243 tablet_service.cc:1467] Processing CreateTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (DEFAULT_TABLE table=test-table [id=a324a2c55145402188500ec25b8a44e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.628111 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.628130 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.628367 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.634678 9432 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Bootstrap starting.
I20260925 01:23:13.635440 9431 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Bootstrap starting.
I20260925 01:23:13.635953 9433 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Bootstrap starting.
I20260925 01:23:13.637173 9431 tablet_bootstrap.cc:654] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.637387 9432 tablet_bootstrap.cc:654] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.637713 9433 tablet_bootstrap.cc:654] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.639531 9431 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: No bootstrap required, opened a new log
I20260925 01:23:13.639530 9432 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: No bootstrap required, opened a new log
I20260925 01:23:13.639670 9432 ts_tablet_manager.cc:1397] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 01:23:13.639671 9431 ts_tablet_manager.cc:1397] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260925 01:23:13.640364 9433 tablet_bootstrap.cc:492] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: No bootstrap required, opened a new log
I20260925 01:23:13.640492 9433 ts_tablet_manager.cc:1397] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 01:23:13.640394 9432 raft_consensus.cc:348] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.640394 9431 raft_consensus.cc:348] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.640635 9432 raft_consensus.cc:374] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.640635 9431 raft_consensus.cc:374] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.640724 9431 raft_consensus.cc:729] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.640724 9432 raft_consensus.cc:729] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.640950 9431 consensus_queue.cc:260] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.640940 9432 consensus_queue.cc:260] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.641135 9433 raft_consensus.cc:348] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.641356 9433 raft_consensus.cc:374] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.641469 9433 raft_consensus.cc:729] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.641486 9431 ts_tablet_manager.cc:1428] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:23:13.641489 9432 ts_tablet_manager.cc:1428] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 01:23:13.641691 9433 consensus_queue.cc:260] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.642524 9433 ts_tablet_manager.cc:1428] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:23:13.674279 9437 raft_consensus.cc:482] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:13.674532 9437 raft_consensus.cc:504] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.676079 9437 leader_election.cc:288] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609), 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:13.689343 9415 raft_consensus.cc:482] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:13.689558 9415 raft_consensus.cc:504] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.690505 9415 leader_election.cc:288] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609)
I20260925 01:23:13.691076 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955" is_pre_election: true
I20260925 01:23:13.691129 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d" is_pre_election: true
I20260925 01:23:13.691308 8990 raft_consensus.cc:2454] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 0.
I20260925 01:23:13.691387 9130 raft_consensus.cc:2454] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 0.
I20260925 01:23:13.691799 9195 leader_election.cc:302] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533a68f6e73f47dfbe860962347fa3bf, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:13.692106 9415 raft_consensus.cc:2781] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:13.692233 9415 raft_consensus.cc:482] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:13.692317 9415 raft_consensus.cc:3037] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.695042 9415 raft_consensus.cc:504] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.696156 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c56c57bff34d1ebd6c53b509fda955"
I20260925 01:23:13.696374 8990 raft_consensus.cc:3037] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.696369 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "533a68f6e73f47dfbe860962347fa3bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d"
I20260925 01:23:13.696492 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.676602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:43911 (local address 127.6.236.131:44293)
0925 01:23:13.688843 (+ 12241us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:13.688848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:13.689451 (+ 603us) server_negotiation.cc:407] Connection header received
0925 01:23:13.689620 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:13.689625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:13.689709 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:13.689808 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:13.690323 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.691513 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.693725 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.694988 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.695171 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.695247 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:13.696053 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 01:23:13.696191 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12103,"thread_start_us":11926,"threads_started":1}
I20260925 01:23:13.696591 9130 raft_consensus.cc:3037] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.696756 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.676454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:47693 (local address 127.6.236.130:42609)
0925 01:23:13.688587 (+ 12133us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:13.688593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:13.689306 (+ 713us) server_negotiation.cc:407] Connection header received
0925 01:23:13.689396 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:13.689400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:13.689481 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:13.689575 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:13.690771 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.691895 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.694260 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.695437 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.695610 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.695683 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:13.696246 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 01:23:13.696396 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11971,"thread_start_us":11802,"threads_started":1}
I20260925 01:23:13.697329 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.676373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44293 (local address 127.6.236.129:43911)
0925 01:23:13.689342 (+ 12969us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:13.689360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:13.689488 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:13.689852 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:13.689856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:13.689885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:13.690131 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:13.690143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.691682 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.691687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:13.693540 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:13.693555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.695190 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.695197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:13.695784 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.695901 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 01:23:13.696725 (+ 824us) client_negotiation.cc:240] Negotiation successful
0925 01:23:13.696859 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12820,"mutex_wait_us":57,"thread_start_us":12364,"threads_started":1}
I20260925 01:23:13.697409 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.676290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.6.236.129:47693)
0925 01:23:13.689078 (+ 12788us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:13.689103 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:13.689266 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:13.690040 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:13.690045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:13.690070 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:13.690545 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:13.690559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.692075 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.692080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:13.694083 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:13.694097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.695651 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.695658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:13.695880 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.695912 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:23:13.695996 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 01:23:13.696147 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12619,"mutex_wait_us":6,"thread_start_us":12733,"threads_started":1}
I20260925 01:23:13.697968 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf" is_pre_election: true
I20260925 01:23:13.698264 9264 raft_consensus.cc:2379] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71c56c57bff34d1ebd6c53b509fda955 in current term 1: Already voted for candidate 533a68f6e73f47dfbe860962347fa3bf in this term.
I20260925 01:23:13.699090 9130 raft_consensus.cc:2454] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 1.
I20260925 01:23:13.699266 8990 raft_consensus.cc:2454] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533a68f6e73f47dfbe860962347fa3bf in term 1.
I20260925 01:23:13.699519 9415 leader_election.cc:288] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 election: Requested vote from peers 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789), 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609)
I20260925 01:23:13.699584 9195 leader_election.cc:302] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf; no voters:
I20260925 01:23:13.699872 9415 raft_consensus.cc:2781] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:13.700039 9415 raft_consensus.cc:686] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 LEADER]: Becoming Leader. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Running, Role: LEADER
I20260925 01:23:13.700062 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9da6c0d46e544d4d94ccdb655ed7e9b5" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d" is_pre_election: true
I20260925 01:23:13.700342 9130 raft_consensus.cc:2379] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71c56c57bff34d1ebd6c53b509fda955 in current term 1: Already voted for candidate 533a68f6e73f47dfbe860962347fa3bf in this term.
I20260925 01:23:13.700349 9415 consensus_queue.cc:237] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.700836 8916 leader_election.cc:302] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71c56c57bff34d1ebd6c53b509fda955; no voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf
I20260925 01:23:13.701254 9437 raft_consensus.cc:2726] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:23:13.703002 8828 catalog_manager.cc:5495] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131). New cstate: current_term: 1 leader_uuid: "533a68f6e73f47dfbe860962347fa3bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } health_report { overall_health: HEALTHY } } }
I20260925 01:23:13.725219 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.720801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.6.236.131:44293)
0925 01:23:13.720988 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:13.720994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:13.721083 (+ 89us) server_negotiation.cc:407] Connection header received
0925 01:23:13.721223 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:13.721226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:13.721288 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:13.721354 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:13.721655 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.722394 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.723741 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.724145 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.724234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.724650 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:13.724747 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:13.724820 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:13.724912 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 01:23:13.724971 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:23:13.733176 8990 raft_consensus.cc:1261] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Refusing update from remote peer 533a68f6e73f47dfbe860962347fa3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:13.733253 9130 raft_consensus.cc:1261] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Refusing update from remote peer 533a68f6e73f47dfbe860962347fa3bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:13.733736 9384 consensus_queue.cc:1035] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:13.734000 9415 consensus_queue.cc:1035] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:13.774829 8827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:13.775054 8827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:13.777812 8827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Sending DeleteTablet for 3 replicas of tablet 9da6c0d46e544d4d94ccdb655ed7e9b5
I20260925 01:23:13.778574 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:48394
I20260925 01:23:13.778554 8966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42462
I20260925 01:23:13.778578 9243 tablet_service.cc:1514] Processing DeleteTablet for tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:13 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:33830
I20260925 01:23:13.778826 9420 tablet_replica.cc:330] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: stopping tablet replica
I20260925 01:23:13.778824 9421 tablet_replica.cc:330] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: stopping tablet replica
I20260925 01:23:13.778848 9422 tablet_replica.cc:330] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: stopping tablet replica
I20260925 01:23:13.779021 9421 raft_consensus.cc:2229] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:13.779011 9420 raft_consensus.cc:2229] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:13.779030 9422 raft_consensus.cc:2229] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:13.779183 9420 raft_consensus.cc:2258] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.779202 9421 raft_consensus.cc:2258] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.779330 9422 raft_consensus.cc:2258] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:13.779780 8827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 01:23:13.780205 9421 ts_tablet_manager.cc:1905] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:13.780251 9422 ts_tablet_manager.cc:1905] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 01:23:13.780400 8827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:13.781381 9420 ts_tablet_manager.cc:1905] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:13.786406 9421 ts_tablet_manager.cc:1918] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:13.786636 9421 log.cc:1196] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/9da6c0d46e544d4d94ccdb655ed7e9b5
I20260925 01:23:13.787042 9422 ts_tablet_manager.cc:1918] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:13.787148 9421 ts_tablet_manager.cc:1939] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 446d9495119a4989b7da34a8b71a2c6d: Deleting consensus metadata
I20260925 01:23:13.787302 9422 log.cc:1196] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/9da6c0d46e544d4d94ccdb655ed7e9b5
I20260925 01:23:13.788154 8813 catalog_manager.cc:4841] TS 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609): tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (table test-table [id=a324a2c55145402188500ec25b8a44e9]) successfully deleted
I20260925 01:23:13.788326 9243 tablet_service.cc:1467] Processing CreateTablet for tablet bb3ce4a610124b7894b159aabed16ab8 (DEFAULT_TABLE table=test-table [id=8e51cb989c9840ce862b69176248b110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.788697 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3ce4a610124b7894b159aabed16ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.788791 8966 tablet_service.cc:1467] Processing CreateTablet for tablet bb3ce4a610124b7894b159aabed16ab8 (DEFAULT_TABLE table=test-table [id=8e51cb989c9840ce862b69176248b110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.789140 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3ce4a610124b7894b159aabed16ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.792701 9431 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Bootstrap starting.
I20260925 01:23:13.793006 9422 ts_tablet_manager.cc:1939] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 533a68f6e73f47dfbe860962347fa3bf: Deleting consensus metadata
I20260925 01:23:13.793278 9433 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Bootstrap starting.
I20260925 01:23:13.793322 9105 tablet_service.cc:1467] Processing CreateTablet for tablet bb3ce4a610124b7894b159aabed16ab8 (DEFAULT_TABLE table=test-table [id=8e51cb989c9840ce862b69176248b110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:13.793679 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3ce4a610124b7894b159aabed16ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:13.793982 8815 catalog_manager.cc:4841] TS 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293): tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (table test-table [id=a324a2c55145402188500ec25b8a44e9]) successfully deleted
I20260925 01:23:13.794318 9431 tablet_bootstrap.cc:654] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.794744 9433 tablet_bootstrap.cc:654] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.797052 9433 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: No bootstrap required, opened a new log
I20260925 01:23:13.797233 9433 ts_tablet_manager.cc:1397] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 01:23:13.798836 9420 ts_tablet_manager.cc:1918] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:13.799080 9420 log.cc:1196] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/9da6c0d46e544d4d94ccdb655ed7e9b5
I20260925 01:23:13.799579 9420 ts_tablet_manager.cc:1939] T 9da6c0d46e544d4d94ccdb655ed7e9b5 P 71c56c57bff34d1ebd6c53b509fda955: Deleting consensus metadata
I20260925 01:23:13.799728 9432 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Bootstrap starting.
I20260925 01:23:13.799373 9433 raft_consensus.cc:348] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.800364 9433 raft_consensus.cc:374] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.800487 9433 raft_consensus.cc:729] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.800755 9433 consensus_queue.cc:260] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.800956 8813 catalog_manager.cc:4841] TS 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789): tablet 9da6c0d46e544d4d94ccdb655ed7e9b5 (table test-table [id=a324a2c55145402188500ec25b8a44e9]) successfully deleted
I20260925 01:23:13.801165 9431 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: No bootstrap required, opened a new log
I20260925 01:23:13.801311 9432 tablet_bootstrap.cc:654] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:13.801339 9431 ts_tablet_manager.cc:1397] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260925 01:23:13.801963 9433 ts_tablet_manager.cc:1428] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 01:23:13.802109 9431 raft_consensus.cc:348] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.802352 9431 raft_consensus.cc:374] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.802435 9431 raft_consensus.cc:729] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a68f6e73f47dfbe860962347fa3bf, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.802675 9431 consensus_queue.cc:260] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.803570 9431 ts_tablet_manager.cc:1428] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:13.805153 9432 tablet_bootstrap.cc:492] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: No bootstrap required, opened a new log
I20260925 01:23:13.805358 9432 ts_tablet_manager.cc:1397] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260925 01:23:13.806398 9432 raft_consensus.cc:348] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.806667 9432 raft_consensus.cc:374] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:13.806804 9432 raft_consensus.cc:729] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d9495119a4989b7da34a8b71a2c6d, State: Initialized, Role: FOLLOWER
I20260925 01:23:13.807062 9432 consensus_queue.cc:260] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.807547 9432 ts_tablet_manager.cc:1428] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:23:13.964452 9437 raft_consensus.cc:482] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:13.964654 9437 raft_consensus.cc:504] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.965274 9437 leader_election.cc:288] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609), 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:13.965842 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3ce4a610124b7894b159aabed16ab8" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf" is_pre_election: true
I20260925 01:23:13.965826 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3ce4a610124b7894b159aabed16ab8" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d" is_pre_election: true
I20260925 01:23:13.966086 9130 raft_consensus.cc:2454] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c56c57bff34d1ebd6c53b509fda955 in term 0.
I20260925 01:23:13.966086 9264 raft_consensus.cc:2454] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c56c57bff34d1ebd6c53b509fda955 in term 0.
I20260925 01:23:13.966568 8918 leader_election.cc:302] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 446d9495119a4989b7da34a8b71a2c6d, 533a68f6e73f47dfbe860962347fa3bf, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:13.966831 9437 raft_consensus.cc:2781] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:13.966936 9437 raft_consensus.cc:482] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:13.967002 9437 raft_consensus.cc:3037] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.969010 9437 raft_consensus.cc:504] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.969524 9437 leader_election.cc:288] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 election: Requested vote from peers 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609), 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293)
I20260925 01:23:13.970000 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3ce4a610124b7894b159aabed16ab8" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533a68f6e73f47dfbe860962347fa3bf"
I20260925 01:23:13.970022 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3ce4a610124b7894b159aabed16ab8" candidate_uuid: "71c56c57bff34d1ebd6c53b509fda955" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446d9495119a4989b7da34a8b71a2c6d"
I20260925 01:23:13.970172 9264 raft_consensus.cc:3037] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.970209 9130 raft_consensus.cc:3037] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:13.972115 9264 raft_consensus.cc:2454] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c56c57bff34d1ebd6c53b509fda955 in term 1.
I20260925 01:23:13.972167 9130 raft_consensus.cc:2454] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c56c57bff34d1ebd6c53b509fda955 in term 1.
I20260925 01:23:13.972481 8918 leader_election.cc:302] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533a68f6e73f47dfbe860962347fa3bf, 71c56c57bff34d1ebd6c53b509fda955; no voters:
I20260925 01:23:13.972764 9437 raft_consensus.cc:2781] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:13.973140 9437 raft_consensus.cc:686] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 LEADER]: Becoming Leader. State: Replica: 71c56c57bff34d1ebd6c53b509fda955, State: Running, Role: LEADER
I20260925 01:23:13.973484 9437 consensus_queue.cc:237] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } }
I20260925 01:23:13.976545 8829 catalog_manager.cc:5495] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "71c56c57bff34d1ebd6c53b509fda955" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71c56c57bff34d1ebd6c53b509fda955" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:14.000034 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:13.994904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.6.236.129:35789)
0925 01:23:13.995483 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:13.995488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:13.995509 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:23:13.995593 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:13.995596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:13.995666 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:13.995798 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:13.996126 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.996879 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.998072 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:13.998480 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:13.998566 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:13.998941 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:13.999512 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:13.999615 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:13.999700 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 01:23:13.999763 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":346,"threads_started":1}
I20260925 01:23:14.013839 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:14.009129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:53515 (local address 127.6.236.130:42609)
0925 01:23:14.009358 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:14.009364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:14.009664 (+ 300us) server_negotiation.cc:407] Connection header received
0925 01:23:14.009802 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:14.009805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:14.009869 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:14.009939 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:14.010355 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.011068 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.012408 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.012825 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.012907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:14.013302 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:14.013406 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:14.013471 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:14.013572 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 01:23:14.013633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:23:14.013936 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:14.008975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42609 (local address 127.6.236.129:53515)
0925 01:23:14.009577 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:14.009593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:14.009705 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:14.009951 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:14.009954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:14.009963 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:14.010213 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:14.010220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.011176 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.011179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:14.012296 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:14.012308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.012959 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.012963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:14.013079 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:14.013159 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:14.013482 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:14.013491 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:23:14.013550 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 01:23:14.013645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
I20260925 01:23:14.021239 9264 raft_consensus.cc:1261] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Refusing update from remote peer 71c56c57bff34d1ebd6c53b509fda955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:14.021360 9130 raft_consensus.cc:1261] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Refusing update from remote peer 71c56c57bff34d1ebd6c53b509fda955: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:14.021802 9452 consensus_queue.cc:1035] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "533a68f6e73f47dfbe860962347fa3bf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:14.022166 9437 consensus_queue.cc:1035] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446d9495119a4989b7da34a8b71a2c6d" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:14.046128 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:14.041300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:35119 (local address 127.6.236.129:35789)
0925 01:23:14.041471 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:14.041476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:14.041719 (+ 243us) server_negotiation.cc:407] Connection header received
0925 01:23:14.041879 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:14.041882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:14.041942 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:14.041997 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:14.042378 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.043099 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.044443 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.045025 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.045130 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:14.045511 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:14.045604 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:14.045673 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:14.045800 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 01:23:14.045874 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:23:14.046216 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:14.041100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:35789 (local address 127.6.236.130:35119)
0925 01:23:14.041624 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:14.041642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:14.041774 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:14.042011 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:14.042014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:14.042022 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:14.042275 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:14.042286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.043228 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.043233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:14.044309 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:14.044323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:14.045186 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:14.045193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:14.045326 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:14.045360 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:14.045690 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:14.045701 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:23:14.045771 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:23:14.045899 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
W20260925 01:23:14.047098 9403 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:23:15.636633 8827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:15.636868 8827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:15.640877 8827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c118c7a23a514196b78d92c2257742e3: Sending DeleteTablet for 3 replicas of tablet bb3ce4a610124b7894b159aabed16ab8
I20260925 01:23:15.641649 8966 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3ce4a610124b7894b159aabed16ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42462
I20260925 01:23:15.641737 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3ce4a610124b7894b159aabed16ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:48394
I20260925 01:23:15.641846 9243 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3ce4a610124b7894b159aabed16ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:33830
I20260925 01:23:15.642472 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8900
I20260925 01:23:15.644675 9488 tablet_replica.cc:330] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: stopping tablet replica
I20260925 01:23:15.644888 9488 raft_consensus.cc:2229] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:15.645116 9488 raft_consensus.cc:2258] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:15.646219 9488 ts_tablet_manager.cc:1905] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:15.653439 9488 ts_tablet_manager.cc:1918] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:23:15.653625 9488 log.cc:1196] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/bb3ce4a610124b7894b159aabed16ab8
I20260925 01:23:15.654075 9488 ts_tablet_manager.cc:1939] T bb3ce4a610124b7894b159aabed16ab8 P 533a68f6e73f47dfbe860962347fa3bf: Deleting consensus metadata
I20260925 01:23:15.655187 9487 tablet_replica.cc:330] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: stopping tablet replica
I20260925 01:23:15.655376 9487 raft_consensus.cc:2229] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:15.655607 9487 raft_consensus.cc:2258] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:15.656548 9487 ts_tablet_manager.cc:1905] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:15.657058 9486 tablet_replica.cc:330] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: stopping tablet replica
I20260925 01:23:15.657259 9486 raft_consensus.cc:2229] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:15.657738 9486 raft_consensus.cc:2258] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:15.658507 9486 ts_tablet_manager.cc:1905] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:15.662544 9487 ts_tablet_manager.cc:1918] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:23:15.662752 9487 log.cc:1196] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/bb3ce4a610124b7894b159aabed16ab8
I20260925 01:23:15.663125 8815 catalog_manager.cc:4841] TS 533a68f6e73f47dfbe860962347fa3bf (127.6.236.131:44293): tablet bb3ce4a610124b7894b159aabed16ab8 (table test-table [id=8e51cb989c9840ce862b69176248b110]) successfully deleted
I20260925 01:23:15.663239 9487 ts_tablet_manager.cc:1939] T bb3ce4a610124b7894b159aabed16ab8 P 446d9495119a4989b7da34a8b71a2c6d: Deleting consensus metadata
I20260925 01:23:15.663554 9486 ts_tablet_manager.cc:1918] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:15.663667 9486 log.cc:1196] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/bb3ce4a610124b7894b159aabed16ab8
I20260925 01:23:15.664013 9486 ts_tablet_manager.cc:1939] T bb3ce4a610124b7894b159aabed16ab8 P 71c56c57bff34d1ebd6c53b509fda955: Deleting consensus metadata
I20260925 01:23:15.664340 8813 catalog_manager.cc:4841] TS 446d9495119a4989b7da34a8b71a2c6d (127.6.236.130:42609): tablet bb3ce4a610124b7894b159aabed16ab8 (table test-table [id=8e51cb989c9840ce862b69176248b110]) successfully deleted
I20260925 01:23:15.664902 8813 catalog_manager.cc:4841] TS 71c56c57bff34d1ebd6c53b509fda955 (127.6.236.129:35789): tablet bb3ce4a610124b7894b159aabed16ab8 (table test-table [id=8e51cb989c9840ce862b69176248b110]) successfully deleted
I20260925 01:23:15.669698 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:15.643747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.6.236.129:35789)
0925 01:23:15.652636 (+ 8889us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:15.652644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:15.653864 (+ 1220us) server_negotiation.cc:407] Connection header received
0925 01:23:15.653934 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:15.653940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:15.654026 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:15.654122 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:15.655942 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.657131 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.658994 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.659587 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.659709 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:15.664841 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:15.664858 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:15.664877 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:15.664917 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:15.667464 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.668070 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.668076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.668079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.668205 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.668484 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.668510 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.668512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.668841 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:15.669042 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:15.669259 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 01:23:15.669420 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8737,"thread_start_us":8618,"threads_started":1}
W20260925 01:23:15.670372 9031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950
W20260925 01:23:15.670933 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:15.671159 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8900
I20260925 01:23:15.690733 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9040
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:15.704934 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:15.691447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.6.236.130:42609)
0925 01:23:15.691950 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:15.691954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:15.691970 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:15.692035 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:15.692039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:15.692103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:15.692193 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:15.693365 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.694072 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.695292 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.695692 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.695789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:15.699713 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:15.699728 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:15.699744 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:15.699781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:15.702689 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.703289 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.703296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.703298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.703431 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.703710 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.703712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.703714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.704056 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:15.704278 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:15.704486 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 01:23:15.704655 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":231,"threads_started":1}
W20260925 01:23:15.705565 9170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44024
W20260925 01:23:15.706163 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9040: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:15.706336 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9040
I20260925 01:23:15.728261 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9179
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299393, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:15.742667 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:15.728946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.6.236.131:44293)
0925 01:23:15.729409 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:15.729414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:15.729430 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:15.729483 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:15.729487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:15.729559 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:15.729647 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:15.730897 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.731618 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.732757 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:15.733142 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:15.733231 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:15.737508 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:15.737524 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:15.737541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:15.737578 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:15.740211 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.740797 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.740804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.740807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.740928 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:15.741286 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:15.741289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:15.741290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:15.741653 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:15.741856 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:15.742121 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 01:23:15.742337 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
W20260925 01:23:15.743317 9309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54090
W20260925 01:23:15.743904 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9179: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:15.744059 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9179
I20260925 01:23:15.762042 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 8797
I20260925 01:23:15.763275 8869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:15.880241 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 8797
2026-09-25T01:23:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8084 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[9496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[9496](info): set up 2 sockets
Sep 25 01:23:15 dist-test-slave-s0b2 krb5kdc[9496](info): commencing operation
krb5kdc: starting...
W20260925 01:23:16.867133 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:16 dist-test-slave-s0b2 krb5kdc[9496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:17.033743 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:34309
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:34309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:17.335922 9509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:17.336185 9509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:17.336311 9509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:17.343250 9509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:17.343358 9509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:17.343434 9509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:17.343513 9509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:17.343571 9509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:17.353068 9509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:34309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:17.354336 9509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:17.355646 9509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:17.366781 9514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:17.366688 9515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:17.366737 9517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:17.367390 9509 server_base.cc:1034] running on GCE node
I20260925 01:23:17.368150 9509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:17.369493 9509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:17.370690 9509 hybrid_clock.cc:648] HybridClock initialized: now 1790299397370642 us; error 37 us; skew 500 ppm
I20260925 01:23:17.371009 9509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:17 dist-test-slave-s0b2 krb5kdc[9496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:17.373884 9509 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:17.375305 9509 webserver.cc:460] Webserver started at http://127.6.236.190:41299/ using document root <none> and password file <none>
I20260925 01:23:17.375958 9509 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:17.376071 9509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:17.376353 9509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:17.379268 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "4ae8431535c345c28f3d628229abb618"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "f197913234415818b0f6848cceaccb87"
server_key_iv: "9e65e99bcb884ab2804608df1992172d"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.380008 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "4ae8431535c345c28f3d628229abb618"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "f197913234415818b0f6848cceaccb87"
server_key_iv: "9e65e99bcb884ab2804608df1992172d"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.384964 9509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:17.388285 9524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:17.389549 9509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:17.389707 9509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "4ae8431535c345c28f3d628229abb618"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "f197913234415818b0f6848cceaccb87"
server_key_iv: "9e65e99bcb884ab2804608df1992172d"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.389884 9509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:17.424009 9509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:17.427948 9509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:17.428208 9509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:17.444809 9509 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:34309
I20260925 01:23:17.444832 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:34309 every 8 connection(s)
I20260925 01:23:17.446126 9509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:17.447582 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9509
I20260925 01:23:17.447810 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:17.448582 7090 external_mini_cluster.cc:1428] Setting key dbbdbb181e6b72329adcaea6e486e1ad
I20260925 01:23:17.450768 9576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:23:17 dist-test-slave-s0b2 krb5kdc[9496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:17.463464 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618: Bootstrap starting.
I20260925 01:23:17.467403 9576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:17.468614 9576 log.cc:824] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:17.471380 9576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618: No bootstrap required, opened a new log
I20260925 01:23:17.471606 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:17.450588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.6.236.190:34309)
0925 01:23:17.451347 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:17.451363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:17.451411 (+ 48us) server_negotiation.cc:407] Connection header received
0925 01:23:17.453513 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:17.453541 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:17.454039 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:17.454500 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:17.455719 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.457744 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.458974 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.459967 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.460162 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:17.463929 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:17.463940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:17.463958 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:17.463999 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:17.467603 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:17.468229 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:17.468237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:17.468245 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:17.468468 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:17.468812 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:17.468817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:17.468819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:17.469470 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:17.469774 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:17.470258 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 01:23:17.470651 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":308,"threads_started":1}
I20260925 01:23:17.479383 9576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } }
I20260925 01:23:17.479727 9576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:17.479812 9576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae8431535c345c28f3d628229abb618, State: Initialized, Role: FOLLOWER
I20260925 01:23:17.480331 9576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } }
I20260925 01:23:17.480542 9576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:17.480656 9576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:17.480806 9576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:17.482987 9576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } }
I20260925 01:23:17.483429 9576 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae8431535c345c28f3d628229abb618; no voters:
I20260925 01:23:17.483949 9576 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:17.484066 9582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:17.484668 9582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae8431535c345c28f3d628229abb618, State: Running, Role: LEADER
I20260925 01:23:17.485137 9582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } }
I20260925 01:23:17.485780 9576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:17.489650 9583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae8431535c345c28f3d628229abb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } } }
I20260925 01:23:17.489951 9583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:17.490398 9584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae8431535c345c28f3d628229abb618. Latest consensus state: current_term: 1 leader_uuid: "4ae8431535c345c28f3d628229abb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8431535c345c28f3d628229abb618" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34309 } } }
I20260925 01:23:17.490674 9584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:17.491910 9591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:17.496532 9591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:17.503963 9591 catalog_manager.cc:1347] Generated new cluster ID: aea9b218577a44c696266e317b016f94
I20260925 01:23:17.504061 9591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:17.520454 9591 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:17.521761 9591 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:17.527702 9591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae8431535c345c28f3d628229abb618: Generated new TSK 0
I20260925 01:23:17.528677 9591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:17.577296 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:17.886175 9603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:17.886449 9603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:17.886587 9603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:17.893910 9603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:17.894018 9603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:17.894166 9603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:17.903610 9603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:17.904768 9603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:17.906006 9603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:17.918545 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:17.918583 9609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:17.918520 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:17.919325 9603 server_base.cc:1034] running on GCE node
I20260925 01:23:17.919888 9603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:17.920747 9603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:17.921940 9603 hybrid_clock.cc:648] HybridClock initialized: now 1790299397921892 us; error 40 us; skew 500 ppm
I20260925 01:23:17.922259 9603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:17 dist-test-slave-s0b2 krb5kdc[9496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:17.925174 9603 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:17.926538 9603 webserver.cc:460] Webserver started at http://127.6.236.129:42781/ using document root <none> and password file <none>
I20260925 01:23:17.927171 9603 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:17.927278 9603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:17.927559 9603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:17.930318 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "6552728aa1824877a8db14d852e5264c"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "1e3853c0b84d67bcab3ecc8ec41e59e9"
server_key_iv: "0dc50d752833fd4b34ffb2179f885dcf"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.931061 9603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "6552728aa1824877a8db14d852e5264c"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "1e3853c0b84d67bcab3ecc8ec41e59e9"
server_key_iv: "0dc50d752833fd4b34ffb2179f885dcf"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.935828 9603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:23:17.939047 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:17.940295 9603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:17.940438 9603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "6552728aa1824877a8db14d852e5264c"
format_stamp: "Formatted at 2026-09-25 01:23:17 on dist-test-slave-s0b2"
server_key: "1e3853c0b84d67bcab3ecc8ec41e59e9"
server_key_iv: "0dc50d752833fd4b34ffb2179f885dcf"
server_key_version: "encryptionkey@0"
I20260925 01:23:17.940618 9603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:17.951459 9603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:17.954959 9603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:17.955204 9603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:17.956063 9603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:17.957271 9603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:17.957340 9603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:17.957450 9603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:17.957500 9603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:17.986222 9603 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:46151
I20260925 01:23:17.986285 9730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:46151 every 8 connection(s)
I20260925 01:23:17.987536 9603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:17.992479 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9603
I20260925 01:23:17.992668 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:17.993338 7090 external_mini_cluster.cc:1428] Setting key 341279ea92674d968114e6a4ee3473c3
Sep 25 01:23:17 dist-test-slave-s0b2 krb5kdc[9496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:18.006703 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:17.989839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55001 (local address 127.6.236.190:34309)
0925 01:23:17.990280 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:17.990287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:17.991185 (+ 898us) server_negotiation.cc:407] Connection header received
0925 01:23:17.993056 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:17.993062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:17.993145 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:17.993247 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:17.994977 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.995984 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.997459 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.998055 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.998175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:18.001373 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:18.001378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:18.001381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:18.001425 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:18.003831 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.004500 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.004504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.004507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.004622 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.005020 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.005023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.005025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.005297 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:18.005437 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:18.006104 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 01:23:18.006352 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20260925 01:23:18.007582 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:17.990297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34309 (local address 127.6.236.129:55001)
0925 01:23:17.991022 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:17.991077 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:17.992792 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:17.993492 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:17.993503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:17.994054 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:17.994814 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:17.994839 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.996134 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.996138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:17.997317 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:17.997329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:17.998278 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:17.998286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:17.998489 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:17.999082 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:17.999104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:18.001166 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:18.003958 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.003967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.003983 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.004380 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.004720 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.004726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.004730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.004909 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.005423 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:18.005430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:18.005747 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 01:23:18.005976 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 01:23:18.006501 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":245,"threads_started":1}
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:23:18.009024 9731 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34309
I20260925 01:23:18.009310 9731 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:18.009975 9731 heartbeater.cc:511] Master 127.6.236.190:34309 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
I20260925 01:23:18.012075 9541 ts_manager.cc:194] Registered new tserver with Master: 6552728aa1824877a8db14d852e5264c (127.6.236.129:46151)
I20260925 01:23:18.013437 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:55001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:18.030557 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:18.333029 9737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:18.333329 9737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:18.333493 9737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:18.340510 9737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:18.340611 9737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:18.340766 9737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:18.350389 9737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:18.351619 9737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:18.352933 9737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:18.370162 9742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:18.370090 9743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:18.370188 9745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:18.371280 9737 server_base.cc:1034] running on GCE node
I20260925 01:23:18.371868 9737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:18.372777 9737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:18.373982 9737 hybrid_clock.cc:648] HybridClock initialized: now 1790299398373939 us; error 31 us; skew 500 ppm
I20260925 01:23:18.374312 9737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:18 dist-test-slave-s0b2 krb5kdc[9496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:18.377233 9737 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:18.378670 9737 webserver.cc:460] Webserver started at http://127.6.236.130:45935/ using document root <none> and password file <none>
I20260925 01:23:18.379324 9737 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:18.379431 9737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:18.379732 9737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:18.382690 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "595e104351864b51ad8ac5c71fc1681c"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "5c113a28ff474a2bad37f792bd70ccfc"
server_key_iv: "dc0cf3d8467ba3bf534d789ea42a62a2"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.383432 9737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "595e104351864b51ad8ac5c71fc1681c"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "5c113a28ff474a2bad37f792bd70ccfc"
server_key_iv: "dc0cf3d8467ba3bf534d789ea42a62a2"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.388545 9737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 01:23:18.391986 9752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.393354 9737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:23:18.393501 9737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "595e104351864b51ad8ac5c71fc1681c"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "5c113a28ff474a2bad37f792bd70ccfc"
server_key_iv: "dc0cf3d8467ba3bf534d789ea42a62a2"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.393687 9737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:18.414577 9737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:18.418699 9737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:18.418960 9737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:18.419870 9737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:18.421123 9737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:18.421196 9737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.421309 9737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:18.421358 9737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.450189 9737 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43809
I20260925 01:23:18.450268 9864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43809 every 8 connection(s)
I20260925 01:23:18.451423 9737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:18.455405 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9737
I20260925 01:23:18.455593 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:18.456017 7090 external_mini_cluster.cc:1428] Setting key 763b1002d56d6001871dddb8975ae6d6
Sep 25 01:23:18 dist-test-slave-s0b2 krb5kdc[9496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:18.472062 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:18.454104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:36387 (local address 127.6.236.190:34309)
0925 01:23:18.454327 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:18.454333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:18.455532 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 01:23:18.458070 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:18.458076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:18.458160 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:18.458293 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:18.460083 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.461068 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.462794 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.463205 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.463297 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:18.466876 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:18.466880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:18.466883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:18.466914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:18.469190 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.469877 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.469880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.469883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.470020 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.470470 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.470474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.470476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.470728 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:18.470856 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:18.471513 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 01:23:18.471733 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:18.473054 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:18.454565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34309 (local address 127.6.236.130:36387)
0925 01:23:18.455364 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:18.455413 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:18.457766 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:18.458512 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:18.458523 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:18.459205 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:18.459896 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:18.459918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.461221 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.461226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:18.462660 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:18.462675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.463411 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.463418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:18.463692 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:18.464325 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:18.464357 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:18.466678 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:18.469317 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.469326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.469343 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.469756 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.470123 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.470128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.470132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.470362 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.470848 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:18.470856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:18.471173 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 01:23:18.471396 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 01:23:18.471906 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":283,"threads_started":1}
I20260925 01:23:18.474372 9865 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:23:18.474704 9865 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:18.475543 9865 heartbeater.cc:511] Master 127.6.236.190:34309 requested a full tablet report, sending...
I20260925 01:23:18.477020 9541 ts_manager.cc:194] Registered new tserver with Master: 595e104351864b51ad8ac5c71fc1681c (127.6.236.130:43809)
I20260925 01:23:18.477780 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:36387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:18.495323 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:18.799312 9870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:18.799575 9870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:18.799796 9870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:18.806874 9870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:18.806977 9870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:18.807132 9870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:18.816454 9870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:18.817636 9870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:18.818907 9870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:18.831400 9878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:18.831364 9876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:18.831429 9875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:18.832046 9870 server_base.cc:1034] running on GCE node
I20260925 01:23:18.832567 9870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:18.833410 9870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:18.834591 9870 hybrid_clock.cc:648] HybridClock initialized: now 1790299398834548 us; error 36 us; skew 500 ppm
I20260925 01:23:18.834874 9870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:18 dist-test-slave-s0b2 krb5kdc[9496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:18.837792 9870 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:18.839169 9870 webserver.cc:460] Webserver started at http://127.6.236.131:41389/ using document root <none> and password file <none>
I20260925 01:23:18.839771 9870 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:18.839865 9870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:18.840118 9870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:18.842846 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "6cd343b539ef404688041c0a44ced924"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "7dd4ddcd244b7162411b136d24832ad4"
server_key_iv: "c1918c9b9c60a8841f8263244b0f671b"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.843592 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "6cd343b539ef404688041c0a44ced924"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "7dd4ddcd244b7162411b136d24832ad4"
server_key_iv: "c1918c9b9c60a8841f8263244b0f671b"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.848580 9870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:18.851867 9885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.853088 9870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:23:18.853224 9870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "6cd343b539ef404688041c0a44ced924"
format_stamp: "Formatted at 2026-09-25 01:23:18 on dist-test-slave-s0b2"
server_key: "7dd4ddcd244b7162411b136d24832ad4"
server_key_iv: "c1918c9b9c60a8841f8263244b0f671b"
server_key_version: "encryptionkey@0"
I20260925 01:23:18.853387 9870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:18.866586 9870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:18.870229 9870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:18.870496 9870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:18.871296 9870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:18.872470 9870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:18.872539 9870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.872623 9870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:18.872669 9870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:18.901962 9870 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:43425
I20260925 01:23:18.901993 9997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:43425 every 8 connection(s)
I20260925 01:23:18.903290 9870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:18.909366 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 9870
I20260925 01:23:18.909567 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:18.910084 7090 external_mini_cluster.cc:1428] Setting key 57fef7e70e615b486b3139470ea900fe
Sep 25 01:23:18 dist-test-slave-s0b2 krb5kdc[9496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:18.922680 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:18.905681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:45423 (local address 127.6.236.190:34309)
0925 01:23:18.905871 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:18.905877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:18.907021 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 01:23:18.908697 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:18.908701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:18.908768 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:18.908850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:18.910727 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.911607 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.913353 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.913784 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.913876 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:18.917147 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:18.917152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:18.917155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:18.917189 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:18.919773 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.920500 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.920504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.920506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.920634 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:18.921031 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:18.921034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:18.921036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:18.921304 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:18.921445 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:18.922089 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 01:23:18.922335 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:23:18.923635 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:18.906161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34309 (local address 127.6.236.131:45423)
0925 01:23:18.906867 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:18.906910 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:18.908457 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:18.909059 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:18.909070 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:18.909755 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:18.910542 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:18.910566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.911758 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.911762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:18.913231 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:18.913242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:18.913954 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:18.913959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:18.914155 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:18.914842 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:18.914863 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:18.916953 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:18.919907 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.919915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.919932 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.920378 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.920735 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:18.920740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:18.920744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:18.920919 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:18.921442 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:18.921449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:18.921758 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 01:23:18.921975 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 01:23:18.922498 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":250,"threads_started":1}
I20260925 01:23:18.924889 9998 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34309
I20260925 01:23:18.925192 9998 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:18.925817 9998 heartbeater.cc:511] Master 127.6.236.190:34309 requested a full tablet report, sending...
I20260925 01:23:18.927155 9541 ts_manager.cc:194] Registered new tserver with Master: 6cd343b539ef404688041c0a44ced924 (127.6.236.131:43425)
I20260925 01:23:18.928004 9541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:45423
I20260925 01:23:18.937196 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:23:18.964779 9733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:18.962393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.6.236.190:34309)
0925 01:23:18.962578 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:18.962585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:18.962968 (+ 383us) server_negotiation.cc:407] Connection header received
0925 01:23:18.963166 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:18.963170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:18.963235 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:18.963316 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:18.964589 (+ 1273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60166: BlockingRecv error: recv got EOF from 127.0.0.1:60166 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:23:18.968266 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9603
W20260925 01:23:18.970570 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9603: Not authorized: Client connection negotiation failed: client connection to 127.6.236.129:46151: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:23:18.970731 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9603
I20260925 01:23:18.983376 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9737
W20260925 01:23:18.985536 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9737: Not authorized: Client connection negotiation failed: client connection to 127.6.236.130:43809: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:23:18.985671 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9737
I20260925 01:23:18.998270 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9870
W20260925 01:23:19.000928 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9870: Not authorized: Client connection negotiation failed: client connection to 127.6.236.131:43425: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:23:19.001048 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9870
I20260925 01:23:19.013806 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 9509
I20260925 01:23:19.014937 9571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:19.116195 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 9509
2026-09-25T01:23:19Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3232 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:19 dist-test-slave-s0b2 krb5kdc[10015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:19 dist-test-slave-s0b2 krb5kdc[10015](info): set up 2 sockets
Sep 25 01:23:19 dist-test-slave-s0b2 krb5kdc[10015](info): commencing operation
krb5kdc: starting...
W20260925 01:23:20.111117 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:20.281332 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40707
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:40707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:20.585213 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:20.585490 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:20.585590 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:20.592803 10028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:20.592904 10028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:20.592957 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:20.593011 10028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:20.593056 10028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:20.603189 10028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:40707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:20.604427 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:20.605850 10028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:20.616712 10033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:20.616742 10036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:20.616789 10034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:20.617373 10028 server_base.cc:1034] running on GCE node
I20260925 01:23:20.618103 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:20.619392 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:20.620612 10028 hybrid_clock.cc:648] HybridClock initialized: now 1790299400620571 us; error 41 us; skew 500 ppm
I20260925 01:23:20.620936 10028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299400, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:20.623898 10028 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:20.625238 10028 webserver.cc:460] Webserver started at http://127.6.236.190:32845/ using document root <none> and password file <none>
I20260925 01:23:20.625852 10028 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:20.625962 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:20.626230 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:20.629009 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "f4147835e73e4efba22c83ce69720087"
format_stamp: "Formatted at 2026-09-25 01:23:20 on dist-test-slave-s0b2"
server_key: "ba0586f50ac9fafe368c5843de3b1b93"
server_key_iv: "724a2fea1521fb10885bd36940952835"
server_key_version: "encryptionkey@0"
I20260925 01:23:20.629740 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "f4147835e73e4efba22c83ce69720087"
format_stamp: "Formatted at 2026-09-25 01:23:20 on dist-test-slave-s0b2"
server_key: "ba0586f50ac9fafe368c5843de3b1b93"
server_key_iv: "724a2fea1521fb10885bd36940952835"
server_key_version: "encryptionkey@0"
I20260925 01:23:20.634405 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260925 01:23:20.637485 10043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:20.638738 10028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:20.638885 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "f4147835e73e4efba22c83ce69720087"
format_stamp: "Formatted at 2026-09-25 01:23:20 on dist-test-slave-s0b2"
server_key: "ba0586f50ac9fafe368c5843de3b1b93"
server_key_iv: "724a2fea1521fb10885bd36940952835"
server_key_version: "encryptionkey@0"
I20260925 01:23:20.639046 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:20.652628 10028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:20.656512 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:20.656752 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:20.673236 10028 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:40707
I20260925 01:23:20.673272 10094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:40707 every 8 connection(s)
I20260925 01:23:20.674538 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:20.676132 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10028
I20260925 01:23:20.676338 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:20.677171 7090 external_mini_cluster.cc:1428] Setting key 902facdf20e3d0d41ca67269f41131b9
I20260925 01:23:20.679067 10095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:20.692168 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087: Bootstrap starting.
I20260925 01:23:20.695442 10095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:20.696705 10095 log.cc:824] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:20.699451 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:20.678980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.6.236.190:40707)
0925 01:23:20.679748 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:20.679760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:20.679798 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:23:20.681309 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:20.681332 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:20.681742 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:20.682070 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:20.683866 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:20.685220 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:20.686962 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:20.687636 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:20.687778 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:20.692391 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:20.692401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:20.692417 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:20.692453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:20.695299 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:20.696130 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:20.696138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:20.696144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:20.696307 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:20.696717 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:20.696720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:20.696722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:20.697359 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:20.697675 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:20.698080 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 01:23:20.698499 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":366,"threads_started":1}
I20260925 01:23:20.699700 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087: No bootstrap required, opened a new log
I20260925 01:23:20.703079 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:23:20.708874 10095 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } }
I20260925 01:23:20.709300 10095 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:20.709399 10095 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4147835e73e4efba22c83ce69720087, State: Initialized, Role: FOLLOWER
I20260925 01:23:20.710000 10095 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } }
I20260925 01:23:20.710301 10095 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:20.710430 10095 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:20.710587 10095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:20.712707 10095 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } }
I20260925 01:23:20.713171 10095 leader_election.cc:302] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4147835e73e4efba22c83ce69720087; no voters:
I20260925 01:23:20.713752 10095 leader_election.cc:288] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:20.713862 10101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:20.714927 10101 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [term 1 LEADER]: Becoming Leader. State: Replica: f4147835e73e4efba22c83ce69720087, State: Running, Role: LEADER
I20260925 01:23:20.715521 10101 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } }
I20260925 01:23:20.716029 10095 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:20.718379 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4147835e73e4efba22c83ce69720087. Latest consensus state: current_term: 1 leader_uuid: "f4147835e73e4efba22c83ce69720087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } } }
I20260925 01:23:20.718755 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:20.719295 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4147835e73e4efba22c83ce69720087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4147835e73e4efba22c83ce69720087" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40707 } } }
I20260925 01:23:20.719579 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:20.728369 10107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:20.733560 10107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:20.743830 10107 catalog_manager.cc:1347] Generated new cluster ID: ebafc796974a4f859d56c7c20bf0cc65
I20260925 01:23:20.743927 10107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:20.752282 10107 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:20.753238 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:20.744622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.6.236.190:40707)
0925 01:23:20.744793 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:20.744799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:20.744820 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:23:20.745052 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:20.745057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:20.745149 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:20.745227 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:20.746548 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:20.747695 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:20.749502 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:20.750169 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:20.750386 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:20.751562 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:20.751569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:20.751574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:20.752370 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:20.752451 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:20.752484 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:20.752589 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:20.752775 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 01:23:20.752878 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:20.753840 10107 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:20.762955 10107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4147835e73e4efba22c83ce69720087: Generated new TSK 0
I20260925 01:23:20.763782 10107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:23:20.771432 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10028
I20260925 01:23:20.772374 10090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:20.859881 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10028
2026-09-25T01:23:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1738 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10129](info): set up 2 sockets
Sep 25 01:23:20 dist-test-slave-s0b2 krb5kdc[10129](info): commencing operation
krb5kdc: starting...
W20260925 01:23:21.926357 7090 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 01:23:22.159138 7090 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.233s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 01:23:22.359498 7090 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.200s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 01:23:22.656675 7090 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.297s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/test-user.keytab.
W20260925 01:23:22.965176 7090 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.308s user 0.000s sys 0.004s
Sep 25 01:23:22 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:23.041136 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:32907
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:32907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:23.349220 10142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:23.349495 10142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:23.349608 10142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:23.356763 10142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:23.356868 10142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:23.356927 10142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:23.357025 10142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:23.357084 10142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:23.366819 10142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:32907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:23.368105 10142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:23.369585 10142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:23.381381 10147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:23.381361 10150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:23.381546 10148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:23.387583 10142 server_base.cc:1034] running on GCE node
I20260925 01:23:23.388357 10142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:23.389724 10142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:23.390913 10142 hybrid_clock.cc:648] HybridClock initialized: now 1790299403390856 us; error 45 us; skew 500 ppm
I20260925 01:23:23.391258 10142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:23 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:23.398766 10142 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:23.400245 10142 webserver.cc:460] Webserver started at http://127.6.236.190:45385/ using document root <none> and password file <none>
I20260925 01:23:23.400908 10142 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:23.401019 10142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:23.401299 10142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:23.404196 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "f33c5c6580ab4ed5ad455d7eb73c2113"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "21441bee185062d4a0e80a203eaa9ffa"
server_key_iv: "db4501ac45f8470948e7d1c9154b3104"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.404902 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "f33c5c6580ab4ed5ad455d7eb73c2113"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "21441bee185062d4a0e80a203eaa9ffa"
server_key_iv: "db4501ac45f8470948e7d1c9154b3104"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.409783 10142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:23:23.413048 10157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:23.414484 10142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:23.414667 10142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "f33c5c6580ab4ed5ad455d7eb73c2113"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "21441bee185062d4a0e80a203eaa9ffa"
server_key_iv: "db4501ac45f8470948e7d1c9154b3104"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.414837 10142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:23.426853 10142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:23.430959 10142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:23.431217 10142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:23.448163 10142 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:32907
I20260925 01:23:23.448210 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:32907 every 8 connection(s)
I20260925 01:23:23.449441 10142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:23.453056 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:23.456687 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10142
I20260925 01:23:23.456869 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:23.457479 7090 external_mini_cluster.cc:1428] Setting key 0b6e31c4327a48fe8ac2200a1480b5d0
I20260925 01:23:23.463652 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113: Bootstrap starting.
I20260925 01:23:23.467060 10209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:23.468143 10209 log.cc:824] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:23.471433 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113: No bootstrap required, opened a new log
Sep 25 01:23:23 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:23.478770 10209 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } }
I20260925 01:23:23.479128 10209 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:23.479215 10209 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33c5c6580ab4ed5ad455d7eb73c2113, State: Initialized, Role: FOLLOWER
I20260925 01:23:23.479550 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:23.459389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.236.190:32907)
0925 01:23:23.460132 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:23.460143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:23.460186 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:23:23.462119 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:23.462141 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:23.462563 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:23.462868 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:23.464526 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:23.465816 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:23.467503 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:23.468154 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:23.468307 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:23.472624 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:23.472632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:23.472646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:23.472684 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:23.475651 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:23.476273 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:23.476280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:23.476285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:23.476447 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:23.476781 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:23.476785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:23.476787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:23.477311 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:23.477628 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:23.478063 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 01:23:23.478463 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":314,"threads_started":1}
I20260925 01:23:23.479777 10209 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } }
I20260925 01:23:23.480005 10209 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:23.480135 10209 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:23.480315 10209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:23.482676 10209 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } }
I20260925 01:23:23.483088 10209 leader_election.cc:302] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33c5c6580ab4ed5ad455d7eb73c2113; no voters:
I20260925 01:23:23.483565 10209 leader_election.cc:288] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:23.483680 10214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:23.484287 10214 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [term 1 LEADER]: Becoming Leader. State: Replica: f33c5c6580ab4ed5ad455d7eb73c2113, State: Running, Role: LEADER
I20260925 01:23:23.484791 10214 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } }
I20260925 01:23:23.485332 10209 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:23.488124 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33c5c6580ab4ed5ad455d7eb73c2113. Latest consensus state: current_term: 1 leader_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } } }
I20260925 01:23:23.488408 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:23.489265 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c5c6580ab4ed5ad455d7eb73c2113" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 32907 } } }
I20260925 01:23:23.489786 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:23.489979 10224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:23.494961 10224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:23.495846 10232 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:23:23.503319 10224 catalog_manager.cc:1347] Generated new cluster ID: 376be1c35ef947a8a8a6aa82e0aee774
I20260925 01:23:23.503429 10224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:23.539732 10224 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:23.541019 10224 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:23.547626 10224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33c5c6580ab4ed5ad455d7eb73c2113: Generated new TSK 0
I20260925 01:23:23.548403 10224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:23.588904 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:23.917196 10236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:23.917457 10236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:23.917547 10236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:23.924682 10236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:23.924782 10236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:23.924901 10236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:23.934258 10236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:23.935490 10236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:23.936776 10236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:23.949007 10242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:23.949120 10241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:23.949105 10244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:23.949926 10236 server_base.cc:1034] running on GCE node
I20260925 01:23:23.950520 10236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:23.951342 10236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:23.952525 10236 hybrid_clock.cc:648] HybridClock initialized: now 1790299403952478 us; error 41 us; skew 500 ppm
I20260925 01:23:23.952783 10236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:23 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:23.955752 10236 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:23.957110 10236 webserver.cc:460] Webserver started at http://127.6.236.129:44333/ using document root <none> and password file <none>
I20260925 01:23:23.957723 10236 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:23.957834 10236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:23.958105 10236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:23.960834 10236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "8f11beb3a6504d7a8ec2315d029e0791"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "2cfb3894945e969a1460af2f8c15eda1"
server_key_iv: "e93036949426a9898b2ae8a4cc968593"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.961565 10236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "8f11beb3a6504d7a8ec2315d029e0791"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "2cfb3894945e969a1460af2f8c15eda1"
server_key_iv: "e93036949426a9898b2ae8a4cc968593"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.966393 10236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:23:23.969688 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:23.970981 10236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:23.971131 10236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "8f11beb3a6504d7a8ec2315d029e0791"
format_stamp: "Formatted at 2026-09-25 01:23:23 on dist-test-slave-s0b2"
server_key: "2cfb3894945e969a1460af2f8c15eda1"
server_key_iv: "e93036949426a9898b2ae8a4cc968593"
server_key_version: "encryptionkey@0"
I20260925 01:23:23.971309 10236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:23.992419 10236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:23.996579 10236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:23.996841 10236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:23.997787 10236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:23.999121 10236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:23.999202 10236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:23.999318 10236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:23.999369 10236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.027928 10236 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:39031
I20260925 01:23:24.027992 10363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:39031 every 8 connection(s)
I20260925 01:23:24.029433 10236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:24.037868 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10236
I20260925 01:23:24.038049 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:24.038739 7090 external_mini_cluster.cc:1428] Setting key 06d112bebe74bcb03e4a8505a63fc78b
Sep 25 01:23:24 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299403, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:24.049131 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.032022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42735 (local address 127.6.236.190:32907)
0925 01:23:24.032450 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:24.032454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:24.033293 (+ 839us) server_negotiation.cc:407] Connection header received
0925 01:23:24.035129 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:24.035134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:24.035212 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:24.035308 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:24.036799 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.037535 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.039162 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.039672 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.039784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.043111 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:24.043117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:24.043121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:24.043164 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:24.045944 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.046864 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.046868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.046870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.046985 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.047452 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.047456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.047458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.047753 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:24.047907 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:24.048573 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 01:23:24.048788 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260925 01:23:24.050143 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.032440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:32907 (local address 127.6.236.129:42735)
0925 01:23:24.033136 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:24.033175 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:24.034862 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:24.035555 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:24.035564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:24.036058 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:24.036630 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.036655 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.037678 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.037682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.039028 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.039041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.039883 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.039892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.040122 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.040776 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:24.040799 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:24.042879 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:24.046085 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.046095 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.046112 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.046737 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.047095 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.047099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.047103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.047330 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.047876 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:24.047884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:24.048219 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 01:23:24.048451 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 01:23:24.048948 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260925 01:23:24.051636 10364 heartbeater.cc:347] Connected to a master server at 127.6.236.190:32907
I20260925 01:23:24.051964 10364 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:24.052821 10364 heartbeater.cc:511] Master 127.6.236.190:32907 requested a full tablet report, sending...
I20260925 01:23:24.054804 10174 ts_manager.cc:194] Registered new tserver with Master: 8f11beb3a6504d7a8ec2315d029e0791 (127.6.236.129:39031)
I20260925 01:23:24.056181 10174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:42735
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:24.079154 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:24.381979 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:24.382289 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:24.382400 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:24.389629 10370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:24.389735 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:24.389854 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:24.399500 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:24.400731 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:24.402004 10370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:24.414258 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:24.414350 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:24.414621 10375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:24.415429 10370 server_base.cc:1034] running on GCE node
I20260925 01:23:24.416132 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:24.417059 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:24.418313 10370 hybrid_clock.cc:648] HybridClock initialized: now 1790299404418251 us; error 35 us; skew 500 ppm
I20260925 01:23:24.418630 10370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:24 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299404, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:24.421689 10370 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:24.423076 10370 webserver.cc:460] Webserver started at http://127.6.236.130:43995/ using document root <none> and password file <none>
I20260925 01:23:24.423722 10370 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:24.423830 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:24.424124 10370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:24.427011 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "56956ef6ee44405999bf1f4072f5e204"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "0a13efadeec4b4b8c1c18fd9bd8b9992"
server_key_iv: "3eb1d7e4a79c18c65e59c732590322a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.427765 10370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "56956ef6ee44405999bf1f4072f5e204"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "0a13efadeec4b4b8c1c18fd9bd8b9992"
server_key_iv: "3eb1d7e4a79c18c65e59c732590322a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.432695 10370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:24.436100 10385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.437371 10370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:24.437523 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "56956ef6ee44405999bf1f4072f5e204"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "0a13efadeec4b4b8c1c18fd9bd8b9992"
server_key_iv: "3eb1d7e4a79c18c65e59c732590322a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.437706 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:24.456363 10370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:24.460294 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:24.460551 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:24.461441 10370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:24.462714 10370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:24.462790 10370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.462911 10370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:24.462971 10370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.491995 10370 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36357
I20260925 01:23:24.492033 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36357 every 8 connection(s)
I20260925 01:23:24.493201 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:24.494027 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10370
I20260925 01:23:24.494201 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:24.494939 7090 external_mini_cluster.cc:1428] Setting key 2039c587c4ee9e92ebeba5f397a1b3b8
Sep 25 01:23:24 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299404, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:24.514878 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.496071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:53273 (local address 127.6.236.190:32907)
0925 01:23:24.496734 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:24.496740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:24.497620 (+ 880us) server_negotiation.cc:407] Connection header received
0925 01:23:24.500033 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:24.500038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:24.500107 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:24.500191 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:24.502089 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.502885 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.504436 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.505068 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.505185 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.508533 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:24.508539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:24.508543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:24.508582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:24.511998 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.512699 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.512704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.512708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.512870 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.513265 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.513270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.513273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.513618 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:24.513778 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:24.514423 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 01:23:24.514644 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260925 01:23:24.515825 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.496581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:32907 (local address 127.6.236.130:53273)
0925 01:23:24.497445 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:24.497501 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:24.499746 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:24.500445 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:24.500456 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:24.501130 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:24.501894 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.501919 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.503037 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.503043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.504294 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.504305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.505264 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.505269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.505455 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.506097 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:24.506123 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:24.508315 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:24.512157 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.512163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.512175 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.512578 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.512973 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.512976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.512978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.513161 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.513730 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:24.513736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:24.514059 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 01:23:24.514318 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 01:23:24.514716 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:23:24.517338 10498 heartbeater.cc:347] Connected to a master server at 127.6.236.190:32907
I20260925 01:23:24.517640 10498 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:24.518443 10498 heartbeater.cc:511] Master 127.6.236.190:32907 requested a full tablet report, sending...
I20260925 01:23:24.519964 10174 ts_manager.cc:194] Registered new tserver with Master: 56956ef6ee44405999bf1f4072f5e204 (127.6.236.130:36357)
I20260925 01:23:24.520761 10174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:53273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:24.538412 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:24.845099 10503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:24.845364 10503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:24.845495 10503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:24.852686 10503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:24.852787 10503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:24.852944 10503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:24.862672 10503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:32907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:24.863962 10503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:24.865243 10503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:24.877683 10509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:24.877707 10511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:24.877637 10508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:24.878504 10503 server_base.cc:1034] running on GCE node
I20260925 01:23:24.879053 10503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:24.879941 10503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:24.881151 10503 hybrid_clock.cc:648] HybridClock initialized: now 1790299404881102 us; error 37 us; skew 500 ppm
I20260925 01:23:24.881453 10503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:24 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299404, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:24.884418 10503 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:24.885797 10503 webserver.cc:460] Webserver started at http://127.6.236.131:42487/ using document root <none> and password file <none>
I20260925 01:23:24.886442 10503 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:24.886545 10503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:24.886818 10503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:24.889569 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "7059cbba18454295993dff12f5c62d9b"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "d1d61eb90dda26494642e9a75bb874ca"
server_key_iv: "6cc09e9dc0ec7c1170bfdd80e358d3e9"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.890275 10503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "7059cbba18454295993dff12f5c62d9b"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "d1d61eb90dda26494642e9a75bb874ca"
server_key_iv: "6cc09e9dc0ec7c1170bfdd80e358d3e9"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.895140 10503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:23:24.898274 10518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.899492 10503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 01:23:24.899631 10503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "7059cbba18454295993dff12f5c62d9b"
format_stamp: "Formatted at 2026-09-25 01:23:24 on dist-test-slave-s0b2"
server_key: "d1d61eb90dda26494642e9a75bb874ca"
server_key_iv: "6cc09e9dc0ec7c1170bfdd80e358d3e9"
server_key_version: "encryptionkey@0"
I20260925 01:23:24.899799 10503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:24.919374 10503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:24.923012 10503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:24.923262 10503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:24.924198 10503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:24.925421 10503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:24.925501 10503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.925581 10503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:24.925623 10503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:24.954813 10503 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:39235
I20260925 01:23:24.954867 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:39235 every 8 connection(s)
I20260925 01:23:24.956037 10503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:24.964322 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10503
I20260925 01:23:24.964500 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:24.964939 7090 external_mini_cluster.cc:1428] Setting key fbfc349327f00c636c68c38d71925ee0
Sep 25 01:23:24 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299404, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:24.975320 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.958548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:49073 (local address 127.6.236.190:32907)
0925 01:23:24.958757 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:24.958764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:24.959961 (+ 1197us) server_negotiation.cc:407] Connection header received
0925 01:23:24.961768 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:24.961772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:24.961844 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:24.961940 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:24.963477 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.964370 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.966220 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.966655 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.966767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.970015 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:24.970020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:24.970023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:24.970062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:24.972492 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.973145 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.973150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.973152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.973265 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:24.973662 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:24.973667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:24.973670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:24.973940 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:24.974077 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:24.974769 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 01:23:24.974980 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:23:24.976256 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:24.959075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:32907 (local address 127.6.236.131:49073)
0925 01:23:24.959791 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:24.959829 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:24.961466 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:24.962142 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:24.962150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:24.962726 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:24.963283 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.963301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.964599 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.964604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.966016 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:24.966031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:24.966872 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:24.966880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:24.967115 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:24.967718 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:24.967745 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:24.969797 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:24.972622 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.972631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.972648 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.973036 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.973362 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:24.973367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:24.973371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:24.973560 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:24.974060 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:24.974068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:24.974432 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 01:23:24.974678 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 01:23:24.975177 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":287,"threads_started":1}
I20260925 01:23:24.977448 10631 heartbeater.cc:347] Connected to a master server at 127.6.236.190:32907
I20260925 01:23:24.977739 10631 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:24.978451 10631 heartbeater.cc:511] Master 127.6.236.190:32907 requested a full tablet report, sending...
I20260925 01:23:24.979904 10173 ts_manager.cc:194] Registered new tserver with Master: 7059cbba18454295993dff12f5c62d9b (127.6.236.131:39235)
I20260925 01:23:24.980775 10173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:49073
I20260925 01:23:24.992252 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299405, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299405, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:25.037411 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:25.023277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.6.236.190:32907)
0925 01:23:25.023435 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:25.023439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:25.023912 (+ 473us) server_negotiation.cc:407] Connection header received
0925 01:23:25.024053 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:25.024057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:25.024141 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:25.024232 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:25.025578 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.026341 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.027600 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.028175 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.028335 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:25.032384 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:25.032389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:25.032393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:25.032426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:25.035033 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.035635 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.035638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.035641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.035772 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.036140 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.036145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.036147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.036513 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:25.036694 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:25.037016 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 01:23:25.037189 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 01:23:25.037984 10173 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38596
I20260925 01:23:25.039556 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10236
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299405, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:25.054342 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:25.040553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.6.236.129:39031)
0925 01:23:25.041179 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:25.041186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:25.041208 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:23:25.041260 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:25.041265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:25.041496 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:25.041632 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:25.042811 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.043787 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.044958 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.045487 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.045597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:25.049253 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:25.049264 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:25.049277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:25.049314 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:25.052082 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.052649 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.052656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.052658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.052782 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.053053 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.053056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.053058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.053382 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:25.053602 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:25.053834 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 01:23:25.054043 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":313,"threads_started":1}
W20260925 01:23:25.055579 10359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41382
W20260925 01:23:25.056699 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10236: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:25.056898 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10236
I20260925 01:23:25.069818 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10370
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299405, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:25.084316 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:25.070601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.6.236.130:36357)
0925 01:23:25.071153 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:25.071159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:25.071185 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:23:25.071250 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:25.071256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:25.071475 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:25.071605 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:25.072750 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.073741 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.074964 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.075565 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.075675 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:25.079320 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:25.079331 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:25.079344 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:25.079383 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:25.081973 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.082652 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.082659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.082661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.082785 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.083094 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.083097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.083099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.083439 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:25.083662 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:25.083841 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 01:23:25.084044 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":308,"threads_started":1}
W20260925 01:23:25.085585 10493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51880
W20260925 01:23:25.086783 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10370: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:25.086920 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10370
I20260925 01:23:25.100111 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10503
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299405, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:25.115221 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:25.100919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.6.236.131:39235)
0925 01:23:25.101520 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:25.101529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:25.101558 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:23:25.101617 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:25.101623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:25.101827 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:25.101960 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:25.103302 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.104435 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.105622 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:25.106155 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:25.106330 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:25.110148 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:25.110160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:25.110176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:25.110207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:25.112773 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.113373 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.113380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.113382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.113488 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:25.113759 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:25.113762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:25.113764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:25.114087 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:25.114367 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:25.114735 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 01:23:25.114926 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":273,"threads_started":1}
W20260925 01:23:25.116442 10626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52348
W20260925 01:23:25.117487 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:23:25.117645 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10503
I20260925 01:23:25.130646 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10142
I20260925 01:23:25.131845 10204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:25.235878 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10142
2026-09-25T01:23:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4382 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10648](info): set up 2 sockets
Sep 25 01:23:25 dist-test-slave-s0b2 krb5kdc[10648](info): commencing operation
krb5kdc: starting...
W20260925 01:23:26.243355 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:26 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:26.421622 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33619
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:26.733022 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:26.733291 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:26.733457 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:26.740641 10661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:26.740736 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:26.740796 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:26.740850 10661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:26.740892 10661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:26.750473 10661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:33619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:26.751662 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:26.752889 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:26.763773 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:26.763811 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:26.763715 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:26.764816 10661 server_base.cc:1034] running on GCE node
I20260925 01:23:26.765484 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:26.766875 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:26.768074 10661 hybrid_clock.cc:648] HybridClock initialized: now 1790299406768017 us; error 36 us; skew 500 ppm
I20260925 01:23:26.768436 10661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:26 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299406, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:26.779884 10661 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:26.782037 10661 webserver.cc:460] Webserver started at http://127.6.236.190:37245/ using document root <none> and password file <none>
I20260925 01:23:26.783094 10661 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:26.783238 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:26.783720 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:26.787730 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "d071efd9ad0f485087b6c05a211d936a"
format_stamp: "Formatted at 2026-09-25 01:23:26 on dist-test-slave-s0b2"
server_key: "6275fa0a0e8d6829c902af9c79966ef7"
server_key_iv: "733a7478dc8dd201b476b4b0b31d55b7"
server_key_version: "encryptionkey@0"
I20260925 01:23:26.788733 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "d071efd9ad0f485087b6c05a211d936a"
format_stamp: "Formatted at 2026-09-25 01:23:26 on dist-test-slave-s0b2"
server_key: "6275fa0a0e8d6829c902af9c79966ef7"
server_key_iv: "733a7478dc8dd201b476b4b0b31d55b7"
server_key_version: "encryptionkey@0"
I20260925 01:23:26.795121 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260925 01:23:26.799399 10676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:26.801990 10661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260925 01:23:26.802199 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "d071efd9ad0f485087b6c05a211d936a"
format_stamp: "Formatted at 2026-09-25 01:23:26 on dist-test-slave-s0b2"
server_key: "6275fa0a0e8d6829c902af9c79966ef7"
server_key_iv: "733a7478dc8dd201b476b4b0b31d55b7"
server_key_version: "encryptionkey@0"
I20260925 01:23:26.802455 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:26.819233 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:26.823006 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:26.823232 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:26.839344 10661 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:33619
I20260925 01:23:26.839377 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:33619 every 8 connection(s)
I20260925 01:23:26.840595 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:26.844177 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:26.848296 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10661
I20260925 01:23:26.848544 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:26.849253 7090 external_mini_cluster.cc:1428] Setting key 485fd02024a74203e32885b653bc44dd
I20260925 01:23:26.855495 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a: Bootstrap starting.
I20260925 01:23:26.858992 10728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:26.860109 10728 log.cc:824] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:26.863269 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a: No bootstrap required, opened a new log
Sep 25 01:23:26 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:26.870821 10728 raft_consensus.cc:348] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } }
I20260925 01:23:26.871162 10728 raft_consensus.cc:374] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:26.871268 10728 raft_consensus.cc:729] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d071efd9ad0f485087b6c05a211d936a, State: Initialized, Role: FOLLOWER
I20260925 01:23:26.871905 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:26.851073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.6.236.190:33619)
0925 01:23:26.851836 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:26.851847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:26.851887 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:23:26.854031 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:26.854054 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:26.854486 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:26.854806 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:26.856579 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:26.857880 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:26.859568 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:26.860259 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:26.860392 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:26.864748 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:26.864756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:26.864773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:26.864816 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:26.867815 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:26.868463 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:26.868469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:26.868475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:26.868655 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:26.869022 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:26.869029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:26.869031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:26.869588 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:26.869897 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:26.870330 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 01:23:26.870723 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":360,"threads_started":1}
I20260925 01:23:26.871853 10728 consensus_queue.cc:260] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } }
I20260925 01:23:26.872073 10728 raft_consensus.cc:388] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:26.872202 10728 raft_consensus.cc:482] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:26.872331 10728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:26.874675 10728 raft_consensus.cc:504] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } }
I20260925 01:23:26.875144 10728 leader_election.cc:302] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d071efd9ad0f485087b6c05a211d936a; no voters:
I20260925 01:23:26.875663 10728 leader_election.cc:288] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:26.875774 10733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:26.876379 10733 raft_consensus.cc:686] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [term 1 LEADER]: Becoming Leader. State: Replica: d071efd9ad0f485087b6c05a211d936a, State: Running, Role: LEADER
I20260925 01:23:26.876859 10733 consensus_queue.cc:237] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } }
I20260925 01:23:26.877511 10728 sys_catalog.cc:564] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:26.879542 10735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d071efd9ad0f485087b6c05a211d936a. Latest consensus state: current_term: 1 leader_uuid: "d071efd9ad0f485087b6c05a211d936a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } } }
I20260925 01:23:26.879509 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d071efd9ad0f485087b6c05a211d936a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d071efd9ad0f485087b6c05a211d936a" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33619 } } }
I20260925 01:23:26.879981 10735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:26.880034 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:26.887339 10740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:26.891697 10740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:26.900419 10740 catalog_manager.cc:1347] Generated new cluster ID: 93fbd669312e4d529b0b3791d3a2c65e
I20260925 01:23:26.900523 10740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:26.921170 10740 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:26.922523 10740 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:26.928227 10740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d071efd9ad0f485087b6c05a211d936a: Generated new TSK 0
I20260925 01:23:26.928987 10740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:26.975131 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:27.286404 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:27.286716 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:27.286826 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:27.293839 10754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:27.293941 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:27.294080 10754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:27.303761 10754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:27.305002 10754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:27.306294 10754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:27.319181 10760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:27.319154 10762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:27.319214 10759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:27.319947 10754 server_base.cc:1034] running on GCE node
I20260925 01:23:27.320550 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:27.321491 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:27.322677 10754 hybrid_clock.cc:648] HybridClock initialized: now 1790299407322630 us; error 39 us; skew 500 ppm
I20260925 01:23:27.322963 10754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:27 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299407, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:27.325846 10754 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:27.327260 10754 webserver.cc:460] Webserver started at http://127.6.236.129:39431/ using document root <none> and password file <none>
I20260925 01:23:27.327908 10754 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:27.328015 10754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:27.328256 10754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:27.331240 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "168a226976de4577afca2977fc73db36"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "ae70a4778d2d626a2f5546a4d654beef"
server_key_iv: "dfbd54556753f2830710233deb223afe"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.331950 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "168a226976de4577afca2977fc73db36"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "ae70a4778d2d626a2f5546a4d654beef"
server_key_iv: "dfbd54556753f2830710233deb223afe"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.336913 10754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:23:27.340188 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.341552 10754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:27.341702 10754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "168a226976de4577afca2977fc73db36"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "ae70a4778d2d626a2f5546a4d654beef"
server_key_iv: "dfbd54556753f2830710233deb223afe"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.341883 10754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:27.360802 10754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:27.365052 10754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:27.365285 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:27.366171 10754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:27.367499 10754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:27.367570 10754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.367663 10754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:27.367707 10754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.397293 10754 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:41327
I20260925 01:23:27.397334 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:41327 every 8 connection(s)
I20260925 01:23:27.398593 10754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:27.401996 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10754
I20260925 01:23:27.402199 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:27.402838 7090 external_mini_cluster.cc:1428] Setting key 845a8e5da7074840057f6c8efc7e94c5
Sep 25 01:23:27 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299407, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:27.418224 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:27.401057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:47045 (local address 127.6.236.190:33619)
0925 01:23:27.401474 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:27.401478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:27.402678 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 01:23:27.404663 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:27.404669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:27.404748 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:27.404838 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:27.406401 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.407237 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.408845 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.409259 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.409367 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:27.412717 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:27.412722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:27.412725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:27.412757 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:27.415315 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:27.416031 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:27.416035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:27.416037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:27.416156 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:27.416564 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:27.416568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:27.416570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:27.416843 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:27.416974 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:27.417652 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 01:23:27.417875 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":189,"threads_started":1}
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:23:27.419277 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:27.401671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33619 (local address 127.6.236.129:47045)
0925 01:23:27.402501 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:27.402548 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:27.404368 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:27.405090 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:27.405098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:27.405607 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:27.406201 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:27.406219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.407371 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.407375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:27.408711 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:27.408724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.409476 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.409494 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:27.409703 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:27.410369 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:27.410402 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:27.412506 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:27.415446 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:27.415455 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:27.415472 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:27.415906 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:27.416256 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:27.416261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:27.416264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:27.416449 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:27.416963 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:27.416970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:27.417307 (+ 337us) client_negotiation.cc:769] Sending connection context
0925 01:23:27.417541 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 01:23:27.418083 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260925 01:23:27.420832 10882 heartbeater.cc:347] Connected to a master server at 127.6.236.190:33619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
I20260925 01:23:27.421146 10882 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:27.421882 10882 heartbeater.cc:511] Master 127.6.236.190:33619 requested a full tablet report, sending...
I20260925 01:23:27.423985 10693 ts_manager.cc:194] Registered new tserver with Master: 168a226976de4577afca2977fc73db36 (127.6.236.129:41327)
I20260925 01:23:27.425417 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:47045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:27.443284 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:27.750089 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:27.750391 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:27.750524 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:27.757810 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:27.757918 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:27.758062 10888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:27.767966 10888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:27.769212 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:27.770531 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:27.782685 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:27.782760 10893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:27.782756 10894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:27.783604 10888 server_base.cc:1034] running on GCE node
I20260925 01:23:27.784148 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:27.785063 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:27.786279 10888 hybrid_clock.cc:648] HybridClock initialized: now 1790299407786209 us; error 33 us; skew 500 ppm
I20260925 01:23:27.786574 10888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:27 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299407, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:27.789585 10888 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:27.790954 10888 webserver.cc:460] Webserver started at http://127.6.236.130:34895/ using document root <none> and password file <none>
I20260925 01:23:27.791605 10888 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:27.791720 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:27.791990 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:27.794855 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "6e1995e73ab040ad82c1b9f0c749daa2"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "6e5c525ccd445fd8966df59a1e5f5b8d"
server_key_iv: "49c44d86cf28a8b752f314c5b53f5ffb"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.795598 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "6e1995e73ab040ad82c1b9f0c749daa2"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "6e5c525ccd445fd8966df59a1e5f5b8d"
server_key_iv: "49c44d86cf28a8b752f314c5b53f5ffb"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.800629 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:27.803963 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.805181 10888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:23:27.805322 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "6e1995e73ab040ad82c1b9f0c749daa2"
format_stamp: "Formatted at 2026-09-25 01:23:27 on dist-test-slave-s0b2"
server_key: "6e5c525ccd445fd8966df59a1e5f5b8d"
server_key_iv: "49c44d86cf28a8b752f314c5b53f5ffb"
server_key_version: "encryptionkey@0"
I20260925 01:23:27.805500 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:27.817443 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:27.821156 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:27.821403 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:27.822331 10888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:27.823542 10888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:27.823616 10888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.823727 10888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:27.823787 10888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:27.856225 10888 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:44443
I20260925 01:23:27.856240 11015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:44443 every 8 connection(s)
I20260925 01:23:27.857496 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:27.859625 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 10888
I20260925 01:23:27.859826 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:27.860378 7090 external_mini_cluster.cc:1428] Setting key 44767876e76e75f2bc47dfb0347571a7
Sep 25 01:23:27 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299407, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:27.878726 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:27.860133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:36779 (local address 127.6.236.190:33619)
0925 01:23:27.860295 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:27.860300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:27.861678 (+ 1378us) server_negotiation.cc:407] Connection header received
0925 01:23:27.863999 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:27.864003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:27.864069 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:27.864147 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:27.865962 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.866725 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.868697 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.869294 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.869424 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:27.872783 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:27.872789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:27.872794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:27.872832 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:27.875708 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:27.876421 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:27.876427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:27.876430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:27.876584 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:27.876968 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:27.876973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:27.876976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:27.877327 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:27.877480 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:27.878139 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 01:23:27.878390 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:23:27.879503 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:27.860688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33619 (local address 127.6.236.130:36779)
0925 01:23:27.861507 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:27.861562 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:27.863714 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:27.864381 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:27.864391 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:27.865053 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:27.865788 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:27.865814 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.866855 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.866861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:27.868554 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:27.868570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:27.869486 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:27.869491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:27.869679 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:27.870366 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:27.870387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:27.872567 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:27.875835 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:27.875841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:27.875853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:27.876310 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:27.876681 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:27.876684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:27.876686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:27.876862 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:27.877443 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:27.877449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:27.877750 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 01:23:27.878009 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 01:23:27.878418 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":321,"threads_started":1}
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:27.880867 11016 heartbeater.cc:347] Connected to a master server at 127.6.236.190:33619
I20260925 01:23:27.881160 11016 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:27.881928 11016 heartbeater.cc:511] Master 127.6.236.190:33619 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:23:27.883451 10693 ts_manager.cc:194] Registered new tserver with Master: 6e1995e73ab040ad82c1b9f0c749daa2 (127.6.236.130:44443)
I20260925 01:23:27.884268 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:36779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:27.903083 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:28.205621 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:28.205889 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:28.206027 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:28.213184 11021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:28.213296 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:28.213451 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:28.222829 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:28.224040 11021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:28.225302 11021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:28.237317 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:28.237280 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:28.237351 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:28.238271 11021 server_base.cc:1034] running on GCE node
I20260925 01:23:28.238803 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:28.239630 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:28.240810 11021 hybrid_clock.cc:648] HybridClock initialized: now 1790299408240765 us; error 39 us; skew 500 ppm
I20260925 01:23:28.241097 11021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:28.244088 11021 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:28.245465 11021 webserver.cc:460] Webserver started at http://127.6.236.131:35885/ using document root <none> and password file <none>
I20260925 01:23:28.246127 11021 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:28.246261 11021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:28.246583 11021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:28.249346 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "a2456398a60f4df08d8d872d93df4714"
format_stamp: "Formatted at 2026-09-25 01:23:28 on dist-test-slave-s0b2"
server_key: "2983bc07d2ad207adde334d99d25d66c"
server_key_iv: "3a13bfc0180006aa63e44f269240daaf"
server_key_version: "encryptionkey@0"
I20260925 01:23:28.250078 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "a2456398a60f4df08d8d872d93df4714"
format_stamp: "Formatted at 2026-09-25 01:23:28 on dist-test-slave-s0b2"
server_key: "2983bc07d2ad207adde334d99d25d66c"
server_key_iv: "3a13bfc0180006aa63e44f269240daaf"
server_key_version: "encryptionkey@0"
I20260925 01:23:28.254850 11021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 01:23:28.258111 11036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:28.259390 11021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:28.259531 11021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "a2456398a60f4df08d8d872d93df4714"
format_stamp: "Formatted at 2026-09-25 01:23:28 on dist-test-slave-s0b2"
server_key: "2983bc07d2ad207adde334d99d25d66c"
server_key_iv: "3a13bfc0180006aa63e44f269240daaf"
server_key_version: "encryptionkey@0"
I20260925 01:23:28.259707 11021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:28.280395 11021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:28.284480 11021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:28.284768 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:28.285669 11021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:28.286958 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:28.287031 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:28.287141 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:28.287189 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:28.314442 11021 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:36469
I20260925 01:23:28.314479 11148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:36469 every 8 connection(s)
I20260925 01:23:28.315678 11021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:28.318375 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11021
I20260925 01:23:28.318547 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:28.318962 7090 external_mini_cluster.cc:1428] Setting key 03a9962df8870a50f7c91ef3b70ffc46
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:28.334487 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.318016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:39885 (local address 127.6.236.190:33619)
0925 01:23:28.318223 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.318229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.319574 (+ 1345us) server_negotiation.cc:407] Connection header received
0925 01:23:28.321758 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.321763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.321841 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.321921 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.323402 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.324116 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.325557 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.325956 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.326071 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.329210 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.329215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.329218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.329250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.331539 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.332191 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.332195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.332197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.332318 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.332824 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.332828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.332830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.333101 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.333229 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.334022 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.334223 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:23:28.335402 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.318606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33619 (local address 127.6.236.131:39885)
0925 01:23:28.319385 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:28.319437 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:28.321468 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:28.322164 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:28.322172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:28.322711 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:28.323248 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:28.323264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.324253 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.324259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:28.325429 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:28.325441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.326155 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.326162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:28.326418 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.326969 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:28.326990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:28.329021 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:28.331659 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:28.331667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:28.331683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:28.332076 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:28.332434 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:28.332438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:28.332440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:28.332706 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:28.333217 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:28.333223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:28.333629 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 01:23:28.333913 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 01:23:28.334327 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":265,"threads_started":1}
I20260925 01:23:28.336560 11149 heartbeater.cc:347] Connected to a master server at 127.6.236.190:33619
I20260925 01:23:28.336799 11149 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:28.337411 11149 heartbeater.cc:511] Master 127.6.236.190:33619 requested a full tablet report, sending...
I20260925 01:23:28.338641 10693 ts_manager.cc:194] Registered new tserver with Master: a2456398a60f4df08d8d872d93df4714 (127.6.236.131:36469)
I20260925 01:23:28.339404 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:39885
I20260925 01:23:28.345691 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:28.392663 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.375661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.6.236.129:41327)
0925 01:23:28.376260 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.376269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.376301 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:23:28.376495 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.376503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.376718 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.376870 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.378840 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.380300 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.382060 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.382952 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.383079 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.387242 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.387252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.387266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.387304 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.390003 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.390663 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.390670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.390672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.390797 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.391158 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.391162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.391164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.391506 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.391788 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.392078 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.392330 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":284,"threads_started":1}
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:28.414631 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.396699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.6.236.190:33619)
0925 01:23:28.396887 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.396892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.397015 (+ 123us) server_negotiation.cc:407] Connection header received
0925 01:23:28.397171 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.397175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.397242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.397319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.398830 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.399540 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.401312 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.401725 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.401812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.410140 (+ 8328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.410146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.410149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.410182 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.412526 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.413059 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.413062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.413065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.413181 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.413436 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.413439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.413441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.413739 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.413919 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.414140 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.414331 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 01:23:28.415185 10693 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57780
I20260925 01:23:28.416023 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10754
I20260925 01:23:28.427549 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.416912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.6.236.129:41327)
0925 01:23:28.417096 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.417102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.417240 (+ 138us) server_negotiation.cc:407] Connection header received
0925 01:23:28.417390 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.417394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.417481 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.417569 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.418712 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.419380 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.420539 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.420944 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.421050 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.423072 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.423077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.423081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.423109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.425247 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.425802 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.425807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.425810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.425922 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.426284 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.426289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.426292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.426542 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.426755 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.427040 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.427242 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:23:28.428107 10877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:28.428982 10882 heartbeater.cc:503] Master 127.6.236.190:33619 was elected leader, sending a full tablet report...
I20260925 01:23:28.508471 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10754
W20260925 01:23:28.522468 10680 connection.cc:535] server connection from 127.6.236.129:47045 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:23:28.522738 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10888
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:28.537772 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.523535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.6.236.130:44443)
0925 01:23:28.524077 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.524084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.524109 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:23:28.524173 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.524179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.524393 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.524524 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.525623 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.526746 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.527993 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.528597 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.528706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.532585 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.532597 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.532611 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.532645 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.535310 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.535900 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.535908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.535910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.536034 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.536341 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.536345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.536347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.536737 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.537115 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.537260 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.537487 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260925 01:23:28.539044 11011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:28.619323 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10888
I20260925 01:23:28.632306 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11021
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[10648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:28.647955 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:28.633358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.6.236.131:36469)
0925 01:23:28.634029 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:28.634035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:28.634063 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:23:28.634118 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:28.634125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:28.634377 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:28.634544 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:28.635722 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.636705 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.637973 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:28.638585 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:28.638674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:28.642588 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:28.642598 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:28.642612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:28.642650 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:28.645206 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.645805 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.645812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.645815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.645956 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:28.646415 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:28.646420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:28.646424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:28.646762 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:28.647049 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:28.647359 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:23:28.647633 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":375,"threads_started":1}
I20260925 01:23:28.649160 11144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:28.730684 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11021
I20260925 01:23:28.743530 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 10661
I20260925 01:23:28.744666 10723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:28.845148 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 10661
2026-09-25T01:23:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3609 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[11168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[11168](info): set up 2 sockets
Sep 25 01:23:28 dist-test-slave-s0b2 krb5kdc[11168](info): commencing operation
krb5kdc: starting...
W20260925 01:23:29.839270 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:29 dist-test-slave-s0b2 krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:30.017287 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37821
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:30.320528 11181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:30.320817 11181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:30.320909 11181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:30.327870 11181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:30.327970 11181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:30.328015 11181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:30.328070 11181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:30.328116 11181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:30.337786 11181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:30.339141 11181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:30.340395 11181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:30.351143 11186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:30.351215 11187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:30.351189 11189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:30.352113 11181 server_base.cc:1034] running on GCE node
I20260925 01:23:30.352840 11181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:30.354223 11181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:30.355518 11181 hybrid_clock.cc:648] HybridClock initialized: now 1790299410355469 us; error 40 us; skew 500 ppm
I20260925 01:23:30.355876 11181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:30 dist-test-slave-s0b2 krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299410, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:30.358857 11181 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:30.360002 11181 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:30.360100 11181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:30.360368 11181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:30.363402 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "31a27a58ed414e6e861ec5e6a91207c5"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "e8fd44f7d2bc37911037c6593cae3e7c"
server_key_iv: "d1245ec521e380fe0afbb98290d9b6f5"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.364140 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "31a27a58ed414e6e861ec5e6a91207c5"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "e8fd44f7d2bc37911037c6593cae3e7c"
server_key_iv: "d1245ec521e380fe0afbb98290d9b6f5"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.369061 11181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:30.372792 11195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:30.374114 11181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:23:30.374292 11181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "31a27a58ed414e6e861ec5e6a91207c5"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "e8fd44f7d2bc37911037c6593cae3e7c"
server_key_iv: "d1245ec521e380fe0afbb98290d9b6f5"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.374491 11181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:30.389858 11181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:30.393870 11181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:30.394090 11181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:30.411120 11181 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:37821
I20260925 01:23:30.411140 11246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:37821 every 8 connection(s)
I20260925 01:23:30.412001 11181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:30.413034 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11181
I20260925 01:23:30.413252 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:30.413851 7090 external_mini_cluster.cc:1428] Setting key c2d76eddf8961dbb3a1dec7316841456
I20260925 01:23:30.416255 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:30.426950 11247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: Bootstrap starting.
Sep 25 01:23:30 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:30.430275 11247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:30.431288 11247 log.cc:824] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:30.434020 11247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: No bootstrap required, opened a new log
I20260925 01:23:30.434998 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:30.415810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.6.236.190:37821)
0925 01:23:30.416671 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:30.416683 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:30.416725 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:23:30.418354 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:30.418377 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:30.418807 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:30.419188 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:30.420685 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:30.421986 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:30.423253 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:30.423981 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:30.424163 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:30.428044 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:30.428051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:30.428065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:30.428105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:30.431022 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:30.431698 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:30.431706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:30.431714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:30.431951 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:30.432329 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:30.432335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:30.432338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:30.432879 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:30.433156 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:30.433593 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 01:23:30.433995 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":389,"threads_started":1}
I20260925 01:23:30.441885 11247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } }
I20260925 01:23:30.442217 11247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:30.442341 11247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a27a58ed414e6e861ec5e6a91207c5, State: Initialized, Role: FOLLOWER
I20260925 01:23:30.442883 11247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } }
I20260925 01:23:30.443109 11247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:30.443241 11247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:30.443397 11247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:30.445621 11247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } }
I20260925 01:23:30.446079 11247 leader_election.cc:302] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a27a58ed414e6e861ec5e6a91207c5; no voters:
I20260925 01:23:30.446689 11247 leader_election.cc:288] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:30.446798 11252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:30.447436 11252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [term 1 LEADER]: Becoming Leader. State: Replica: 31a27a58ed414e6e861ec5e6a91207c5, State: Running, Role: LEADER
I20260925 01:23:30.447970 11252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } }
I20260925 01:23:30.448683 11247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:30.450781 11254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a27a58ed414e6e861ec5e6a91207c5. Latest consensus state: current_term: 1 leader_uuid: "31a27a58ed414e6e861ec5e6a91207c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } } }
I20260925 01:23:30.450975 11253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31a27a58ed414e6e861ec5e6a91207c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a27a58ed414e6e861ec5e6a91207c5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37821 } } }
I20260925 01:23:30.451481 11253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:30.451681 11254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:30.455341 11260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:30.459633 11260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:30.467597 11260 catalog_manager.cc:1347] Generated new cluster ID: 3f571fbf00124bfa89ebf76be5c2928c
I20260925 01:23:30.467695 11260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:30.484088 11260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:30.485328 11260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:30.500063 11260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: Generated new TSK 0
I20260925 01:23:30.501148 11260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:30.558132 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:30.876019 11273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:30.876273 11273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:30.876392 11273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:30.883657 11273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:30.883761 11273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:30.883913 11273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:30.893352 11273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:30.894644 11273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:30.895924 11273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:30.908128 11278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:30.908164 11279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:30.908205 11281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:30.908922 11273 server_base.cc:1034] running on GCE node
I20260925 01:23:30.909476 11273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:30.910408 11273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:30.911610 11273 hybrid_clock.cc:648] HybridClock initialized: now 1790299410911575 us; error 58 us; skew 500 ppm
I20260925 01:23:30.911912 11273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:30 dist-test-slave-s0b2 krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299410, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:30.914868 11273 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:30.915966 11273 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:30.916064 11273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:30.916356 11273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:30.919261 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "fc09ac528eb24fa4b669b9cea355734a"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "99451d37946dc5446dd8f8b4127da012"
server_key_iv: "94235a6008a930bfecb0327d03d153c8"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.920039 11273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "fc09ac528eb24fa4b669b9cea355734a"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "99451d37946dc5446dd8f8b4127da012"
server_key_iv: "94235a6008a930bfecb0327d03d153c8"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.924870 11273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:30.928117 11287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:30.929294 11273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:30.929441 11273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "fc09ac528eb24fa4b669b9cea355734a"
format_stamp: "Formatted at 2026-09-25 01:23:30 on dist-test-slave-s0b2"
server_key: "99451d37946dc5446dd8f8b4127da012"
server_key_iv: "94235a6008a930bfecb0327d03d153c8"
server_key_version: "encryptionkey@0"
I20260925 01:23:30.929654 11273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:30.954912 11273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:30.958830 11273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:30.959065 11273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:30.959969 11273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:30.961201 11273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:30.961298 11273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:30.961395 11273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:30.961483 11273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:30.991210 11273 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:36943
I20260925 01:23:30.991295 11399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:36943 every 8 connection(s)
I20260925 01:23:30.992030 11273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:31.001641 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11273
I20260925 01:23:31.001861 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:31.002707 7090 external_mini_cluster.cc:1428] Setting key b36f371dbe47ef6e47f2d29e38578a38
Sep 25 01:23:30 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299410, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:31.012207 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:30.994393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:40169 (local address 127.6.236.190:37821)
0925 01:23:30.994810 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:30.994814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:30.995714 (+ 900us) server_negotiation.cc:407] Connection header received
0925 01:23:30.997544 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:30.997550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:30.997630 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:30.997725 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:30.999314 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.000043 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.001673 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.002326 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.002460 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.006363 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:31.006369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:31.006373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:31.006412 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:31.009281 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.009948 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.009953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.009957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.010103 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.010504 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.010508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.010511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.010819 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:31.010971 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:31.011778 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 01:23:31.011938 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":223,"threads_started":1}
I20260925 01:23:31.013552 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:30.994894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37821 (local address 127.6.236.129:40169)
0925 01:23:30.995563 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:30.995607 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:30.997261 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:30.997961 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:30.997969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:30.998550 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:30.999150 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:30.999168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.000224 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.000229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.001525 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:31.001538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.002545 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.002551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.002829 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.003537 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:31.003566 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:31.006121 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:31.009410 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.009417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.009430 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.009845 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.010198 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.010202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.010204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.010408 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.010964 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:31.010973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:31.011394 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 01:23:31.011689 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 01:23:31.012213 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":271,"threads_started":1}
I20260925 01:23:31.015025 11400 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37821
I20260925 01:23:31.015339 11400 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:31.016047 11400 heartbeater.cc:511] Master 127.6.236.190:37821 requested a full tablet report, sending...
I20260925 01:23:31.018110 11212 ts_manager.cc:194] Registered new tserver with Master: fc09ac528eb24fa4b669b9cea355734a (127.6.236.129:36943)
I20260925 01:23:31.019578 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:40169
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:31.045461 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:31.350680 11406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:31.350955 11406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:31.351094 11406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:31.358306 11406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:31.358422 11406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:31.358573 11406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:31.368162 11406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:31.369413 11406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:31.370745 11406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:31.383162 11414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:31.383244 11411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:31.383209 11412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:31.384111 11406 server_base.cc:1034] running on GCE node
I20260925 01:23:31.384678 11406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:31.385617 11406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:31.386801 11406 hybrid_clock.cc:648] HybridClock initialized: now 1790299411386741 us; error 63 us; skew 500 ppm
I20260925 01:23:31.387102 11406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:31 dist-test-slave-s0b2 krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299411, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:31.390029 11406 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:31.391156 11406 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:31.391256 11406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:31.391549 11406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:31.394379 11406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "e36c3c425129465fb1cd5aa4e068d3ca"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "403ae62f48054b07d6afcc1b317dec92"
server_key_iv: "e43a8ecf884105cef8a82042aa4b17b4"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.395162 11406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "e36c3c425129465fb1cd5aa4e068d3ca"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "403ae62f48054b07d6afcc1b317dec92"
server_key_iv: "e43a8ecf884105cef8a82042aa4b17b4"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.399935 11406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:23:31.403303 11420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.404579 11406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:23:31.404726 11406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "e36c3c425129465fb1cd5aa4e068d3ca"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "403ae62f48054b07d6afcc1b317dec92"
server_key_iv: "e43a8ecf884105cef8a82042aa4b17b4"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.404918 11406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:31.436576 11406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:31.440505 11406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:31.440758 11406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:31.441740 11406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:31.443097 11406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:31.443193 11406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.443307 11406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:31.443382 11406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.473124 11406 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:42041
I20260925 01:23:31.473178 11532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:42041 every 8 connection(s)
I20260925 01:23:31.473987 11406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:31.482595 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11406
I20260925 01:23:31.482772 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:31.483237 7090 external_mini_cluster.cc:1428] Setting key 6a10cc05622f612dfc85e6311b57c6b8
Sep 25 01:23:31 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299411, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:31.493922 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:31.476445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:46875 (local address 127.6.236.190:37821)
0925 01:23:31.476624 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:31.476628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:31.477765 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 01:23:31.479604 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:31.479608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:31.479708 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:31.479795 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:31.481319 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.482089 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.483711 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.484232 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.484334 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.487796 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:31.487801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:31.487806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:31.487843 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:31.490755 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.491614 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.491620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.491624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.491778 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.492168 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.492172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.492175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.492526 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:31.492652 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:31.493332 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 01:23:31.493589 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:23:31.494871 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:31.476853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37821 (local address 127.6.236.130:46875)
0925 01:23:31.477582 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:31.477641 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:31.479353 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:31.480055 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:31.480064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:31.480572 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:31.481143 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:31.481161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.482208 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.482212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.483566 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:31.483578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.484449 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.484457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.484719 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.485424 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:31.485490 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:31.487575 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:31.490907 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.490915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.490944 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.491480 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.491877 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.491880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.491882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.492067 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.492629 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:31.492635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:31.492960 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 01:23:31.493202 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 01:23:31.493688 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":282,"threads_started":1}
I20260925 01:23:31.496238 11533 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37821
I20260925 01:23:31.496551 11533 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:31.497169 11533 heartbeater.cc:511] Master 127.6.236.190:37821 requested a full tablet report, sending...
I20260925 01:23:31.498620 11212 ts_manager.cc:194] Registered new tserver with Master: e36c3c425129465fb1cd5aa4e068d3ca (127.6.236.130:42041)
I20260925 01:23:31.499392 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:46875
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:31.525455 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:31.831670 11538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:31.831947 11538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:31.832042 11538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:31.839111 11538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:31.839217 11538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:31.839330 11538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:31.848830 11538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:31.850263 11538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:31.851583 11538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:31.864073 11544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:31.864231 11546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:31.864719 11543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:31.865633 11538 server_base.cc:1034] running on GCE node
I20260925 01:23:31.866318 11538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:31.867254 11538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:31.868455 11538 hybrid_clock.cc:648] HybridClock initialized: now 1790299411868391 us; error 54 us; skew 500 ppm
I20260925 01:23:31.868749 11538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:31 dist-test-slave-s0b2 krb5kdc[11168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299411, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:31.871614 11538 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:31.872681 11538 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:31.872780 11538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:31.873065 11538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:31.875865 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "28eff00af3b3f5216947f423ea5de237"
server_key_iv: "935f8659378eb4c14d311bbab6a104f2"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.876634 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "28eff00af3b3f5216947f423ea5de237"
server_key_iv: "935f8659378eb4c14d311bbab6a104f2"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.881590 11538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:23:31.884972 11552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.886228 11538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:31.886411 11538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0"
format_stamp: "Formatted at 2026-09-25 01:23:31 on dist-test-slave-s0b2"
server_key: "28eff00af3b3f5216947f423ea5de237"
server_key_iv: "935f8659378eb4c14d311bbab6a104f2"
server_key_version: "encryptionkey@0"
I20260925 01:23:31.886634 11538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:31.912890 11538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:31.916980 11538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:31.917235 11538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:31.918233 11538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:31.919538 11538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:31.919621 11538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.919729 11538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:31.919793 11538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:31.947705 11538 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:38211
I20260925 01:23:31.947770 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:38211 every 8 connection(s)
I20260925 01:23:31.948498 11538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:31.952183 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11538
I20260925 01:23:31.952378 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:31.952931 7090 external_mini_cluster.cc:1428] Setting key 02c5da20d999df0b436dde09c077c81d
Sep 25 01:23:31 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299411, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:31.967756 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:31.950977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:34545 (local address 127.6.236.190:37821)
0925 01:23:31.951157 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:31.951161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:31.952507 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 01:23:31.954691 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:31.954697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:31.954784 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:31.954907 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:31.956632 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.957346 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.958885 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.959272 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.959360 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.962440 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:31.962445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:31.962448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:31.962480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:31.964825 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.965485 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.965488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.965491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.965601 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.966002 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.966005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.966015 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.966296 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:31.966409 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:31.967238 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 01:23:31.967414 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:23:31.968853 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:31.951470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37821 (local address 127.6.236.131:34545)
0925 01:23:31.952338 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:31.952398 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:31.954393 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:31.955203 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:31.955212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:31.955785 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:31.956476 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:31.956500 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.957486 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.957490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.958778 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:31.958790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.959432 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.959436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:31.959619 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.960186 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:31.960207 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:31.962266 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:31.964962 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.964968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.964988 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.965394 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.965714 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:31.965718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:31.965720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:31.965911 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:31.966419 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:31.966426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:31.966813 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 01:23:31.967107 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 01:23:31.967623 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":320,"threads_started":1}
I20260925 01:23:31.970103 11665 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37821
I20260925 01:23:31.970410 11665 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:31.971047 11665 heartbeater.cc:511] Master 127.6.236.190:37821 requested a full tablet report, sending...
I20260925 01:23:31.972340 11212 ts_manager.cc:194] Registered new tserver with Master: 29f51a07e5bf47cd8b7f2da048ed1cb0 (127.6.236.131:38211)
I20260925 01:23:31.973129 11212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:34545
I20260925 01:23:31.980271 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:23:31.997404 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:31.984561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.6.236.190:37821)
0925 01:23:31.984759 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:31.984765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:31.985222 (+ 457us) server_negotiation.cc:407] Connection header received
0925 01:23:31.985350 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:31.985354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:31.985413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:31.985485 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:31.986840 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.987594 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.988752 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:31.989174 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:31.989273 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:31.991663 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:31.991671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:31.991675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:31.991715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:31.994775 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.995449 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.995454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.995458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.995612 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:31.996007 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:31.996011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:31.996014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:31.996432 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:31.996667 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:31.996972 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 01:23:31.997129 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:23:32.001405 11212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:32.008100 11212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:32.023003 11400 heartbeater.cc:503] Master 127.6.236.190:37821 was elected leader, sending a full tablet report...
I20260925 01:23:32.029618 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.021406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.6.236.130:42041)
0925 01:23:32.022188 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.022197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.022221 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:23:32.022340 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.022347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.022602 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.023348 (+ 746us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.023573 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025050 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.027215 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.028253 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.028379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.028572 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.029156 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.029311 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":1002,"threads_started":1}
I20260925 01:23:32.029623 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.021026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42041 (local address 127.0.0.1:59972)
0925 01:23:32.021775 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.021977 (+ 202us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.022080 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.022834 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.022842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.022870 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.023302 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.023314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025207 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.025213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.027072 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.027084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.028701 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.028708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.028893 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.028947 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.029082 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.029264 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":320,"threads_started":1}
I20260925 01:23:32.030900 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.020667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:36943 (local address 127.0.0.1:60812)
0925 01:23:32.021776 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.021815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.022062 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.023052 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.023056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.023078 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.023774 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.023790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025882 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.025887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.027978 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.027990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.030112 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.030118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.030318 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.030345 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.030411 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.030520 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":185,"thread_start_us":899,"threads_started":1}
I20260925 01:23:32.030944 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.019857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.6.236.131:38211)
0925 01:23:32.020664 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.020673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.021922 (+ 1249us) server_negotiation.cc:407] Connection header received
0925 01:23:32.022444 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.022451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.022635 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.022768 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.024055 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025467 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.028108 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.029497 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.029616 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.029825 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.030441 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.030611 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":475,"threads_started":1}
I20260925 01:23:32.031076 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.020606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.6.236.129:36943)
0925 01:23:32.021163 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.021173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.021917 (+ 744us) server_negotiation.cc:407] Connection header received
0925 01:23:32.022343 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.022351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.022605 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.022741 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.023969 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025754 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.028144 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.029914 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.030063 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.030291 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.030633 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.030793 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":240,"threads_started":1}
I20260925 01:23:32.031507 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.019785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:38211 (local address 127.0.0.1:43002)
0925 01:23:32.021775 (+ 1990us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.021822 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.022284 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.023077 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.023086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.023110 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.023808 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.023824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.025598 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.025604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.027765 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.027779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.030045 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.030055 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.030225 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.030297 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.030387 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.030520 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"mutex_wait_us":9,"thread_start_us":384,"threads_started":1}
I20260925 01:23:32.033689 11468 tablet_service.cc:1467] Processing CreateTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 (DEFAULT_TABLE table=test-table [id=d297e7c4b3434f2699e647e5cd09abd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:32.034889 11468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc02414e6ffc4cad807fe56d2146a6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:32.034945 11600 tablet_service.cc:1467] Processing CreateTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 (DEFAULT_TABLE table=test-table [id=d297e7c4b3434f2699e647e5cd09abd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:32.034945 11335 tablet_service.cc:1467] Processing CreateTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 (DEFAULT_TABLE table=test-table [id=d297e7c4b3434f2699e647e5cd09abd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:32.036219 11335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc02414e6ffc4cad807fe56d2146a6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:32.036233 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc02414e6ffc4cad807fe56d2146a6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:32.045722 11683 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Bootstrap starting.
I20260925 01:23:32.048702 11683 tablet_bootstrap.cc:654] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:32.050655 11685 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Bootstrap starting.
I20260925 01:23:32.051020 11683 log.cc:824] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:32.051076 11684 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Bootstrap starting.
I20260925 01:23:32.054077 11683 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: No bootstrap required, opened a new log
I20260925 01:23:32.054111 11684 tablet_bootstrap.cc:654] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:32.054382 11683 ts_tablet_manager.cc:1397] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260925 01:23:32.054716 11685 tablet_bootstrap.cc:654] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:32.055130 11684 log.cc:824] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:32.055853 11685 log.cc:824] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:32.057680 11684 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: No bootstrap required, opened a new log
I20260925 01:23:32.057904 11684 ts_tablet_manager.cc:1397] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 01:23:32.058784 11685 tablet_bootstrap.cc:492] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: No bootstrap required, opened a new log
I20260925 01:23:32.059058 11685 ts_tablet_manager.cc:1397] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:23:32.064041 11683 raft_consensus.cc:348] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.064456 11683 raft_consensus.cc:374] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:32.064594 11683 raft_consensus.cc:729] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36c3c425129465fb1cd5aa4e068d3ca, State: Initialized, Role: FOLLOWER
I20260925 01:23:32.065148 11684 raft_consensus.cc:348] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.065209 11683 consensus_queue.cc:260] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.065442 11684 raft_consensus.cc:374] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:32.065548 11684 raft_consensus.cc:729] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc09ac528eb24fa4b669b9cea355734a, State: Initialized, Role: FOLLOWER
I20260925 01:23:32.066319 11533 heartbeater.cc:503] Master 127.6.236.190:37821 was elected leader, sending a full tablet report...
I20260925 01:23:32.066938 11683 ts_tablet_manager.cc:1428] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 01:23:32.067374 11684 consensus_queue.cc:260] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.068641 11685 raft_consensus.cc:348] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.068933 11685 raft_consensus.cc:374] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:32.069022 11685 raft_consensus.cc:729] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f51a07e5bf47cd8b7f2da048ed1cb0, State: Initialized, Role: FOLLOWER
I20260925 01:23:32.069620 11685 consensus_queue.cc:260] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.069934 11684 ts_tablet_manager.cc:1428] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 01:23:32.071611 11665 heartbeater.cc:503] Master 127.6.236.190:37821 was elected leader, sending a full tablet report...
I20260925 01:23:32.072001 11685 ts_tablet_manager.cc:1428] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260925 01:23:32.116986 11691 raft_consensus.cc:482] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:32.117292 11691 raft_consensus.cc:504] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.119263 11691 leader_election.cc:288] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e36c3c425129465fb1cd5aa4e068d3ca (127.6.236.130:42041), fc09ac528eb24fa4b669b9cea355734a (127.6.236.129:36943)
I20260925 01:23:32.124923 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.119690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:52261 (local address 127.6.236.130:42041)
0925 01:23:32.119912 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.119916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.120291 (+ 375us) server_negotiation.cc:407] Connection header received
0925 01:23:32.120478 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.120482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.120542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.120612 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.120987 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.121685 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.123303 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.124083 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.124219 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.124294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.124554 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.124676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 01:23:32.124931 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.119614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42041 (local address 127.6.236.131:52261)
0925 01:23:32.120205 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.120220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.120347 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.120648 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.120653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.120684 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.120853 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.120862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.121808 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.121814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.123147 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.123159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.124219 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.124225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.124400 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.124424 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.124508 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.124636 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":20}
I20260925 01:23:32.125573 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.119931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:49457 (local address 127.6.236.129:36943)
0925 01:23:32.120118 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.120123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.120770 (+ 647us) server_negotiation.cc:407] Connection header received
0925 01:23:32.120928 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.120932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.120989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.121070 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.121585 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.122307 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.123913 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.124683 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.124818 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.124884 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.125215 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.125341 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:23:32.125551 11690 raft_consensus.cc:482] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:32.126070 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.119672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:36943 (local address 127.6.236.131:49457)
0925 01:23:32.120674 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.120688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.120813 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.121096 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.121100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.121122 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.121413 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.121421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.122443 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.122448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.123768 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.123779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.124873 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.124878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.125026 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.125057 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.125599 (+ 542us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.125729 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":1,"thread_start_us":717,"threads_started":1}
I20260925 01:23:32.125864 11690 raft_consensus.cc:504] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.125975 11488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" is_pre_election: true
I20260925 01:23:32.126508 11488 raft_consensus.cc:2454] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f51a07e5bf47cd8b7f2da048ed1cb0 in term 0.
I20260925 01:23:32.127068 11355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc09ac528eb24fa4b669b9cea355734a" is_pre_election: true
I20260925 01:23:32.127285 11555 leader_election.cc:302] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f51a07e5bf47cd8b7f2da048ed1cb0, e36c3c425129465fb1cd5aa4e068d3ca; no voters:
I20260925 01:23:32.127447 11355 raft_consensus.cc:2454] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f51a07e5bf47cd8b7f2da048ed1cb0 in term 0.
I20260925 01:23:32.127676 11691 raft_consensus.cc:2781] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:32.127822 11691 raft_consensus.cc:482] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:32.127969 11691 raft_consensus.cc:3037] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:32.127988 11690 leader_election.cc:288] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f51a07e5bf47cd8b7f2da048ed1cb0 (127.6.236.131:38211), e36c3c425129465fb1cd5aa4e068d3ca (127.6.236.130:42041)
I20260925 01:23:32.130914 11691 raft_consensus.cc:504] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.131552 11691 leader_election.cc:288] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [CANDIDATE]: Term 1 election: Requested vote from peers e36c3c425129465fb1cd5aa4e068d3ca (127.6.236.130:42041), fc09ac528eb24fa4b669b9cea355734a (127.6.236.129:36943)
I20260925 01:23:32.132069 11488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36c3c425129465fb1cd5aa4e068d3ca"
I20260925 01:23:32.132266 11488 raft_consensus.cc:3037] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:32.132266 11355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc09ac528eb24fa4b669b9cea355734a"
I20260925 01:23:32.132470 11355 raft_consensus.cc:3037] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:32.135514 11488 raft_consensus.cc:2454] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f51a07e5bf47cd8b7f2da048ed1cb0 in term 1.
I20260925 01:23:32.135697 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.128326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55567 (local address 127.6.236.131:38211)
0925 01:23:32.128483 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.128494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.128897 (+ 403us) server_negotiation.cc:407] Connection header received
0925 01:23:32.129150 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.129155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.129225 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.129282 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.130010 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.130957 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.133344 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.134562 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.134684 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.134751 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.135334 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.135474 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:23:32.135994 11555 leader_election.cc:302] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f51a07e5bf47cd8b7f2da048ed1cb0, e36c3c425129465fb1cd5aa4e068d3ca; no voters:
I20260925 01:23:32.136368 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.128220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:38211 (local address 127.6.236.129:55567)
0925 01:23:32.128811 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.128825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.129003 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.129477 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.129482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.129512 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.129878 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.129897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.131066 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.131071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.133170 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.133184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.134992 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.134998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.135163 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.135196 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.135830 (+ 634us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.135966 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":2,"thread_start_us":325,"threads_started":1}
I20260925 01:23:32.136410 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.129052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:60681 (local address 127.6.236.130:42041)
0925 01:23:32.129200 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.129204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.129752 (+ 548us) server_negotiation.cc:407] Connection header received
0925 01:23:32.129900 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.129905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.129972 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.130064 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:32.130615 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.131592 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.134101 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.135337 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.135469 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.135549 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.136063 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.136180 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:32.136467 11694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.128949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42041 (local address 127.6.236.129:60681)
0925 01:23:32.129649 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:32.129664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:32.129782 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:32.130083 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:32.130087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:32.130112 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:32.130457 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.130469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.131752 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.131757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.133922 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:32.133936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.135741 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.135748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:32.135904 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.135936 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:23:32.136017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:23:32.136141 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":380,"threads_started":1}
I20260925 01:23:32.136425 11355 raft_consensus.cc:2454] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f51a07e5bf47cd8b7f2da048ed1cb0 in term 1.
I20260925 01:23:32.136675 11691 raft_consensus.cc:2781] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:32.137360 11488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "fc09ac528eb24fa4b669b9cea355734a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" is_pre_election: true
I20260925 01:23:32.137607 11691 raft_consensus.cc:686] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 29f51a07e5bf47cd8b7f2da048ed1cb0, State: Running, Role: LEADER
I20260925 01:23:32.137739 11488 raft_consensus.cc:2379] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc09ac528eb24fa4b669b9cea355734a in current term 1: Already voted for candidate 29f51a07e5bf47cd8b7f2da048ed1cb0 in this term.
I20260925 01:23:32.137784 11620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc02414e6ffc4cad807fe56d2146a6d2" candidate_uuid: "fc09ac528eb24fa4b669b9cea355734a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" is_pre_election: true
I20260925 01:23:32.138262 11691 consensus_queue.cc:237] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } }
I20260925 01:23:32.138784 11290 leader_election.cc:302] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc09ac528eb24fa4b669b9cea355734a; no voters: 29f51a07e5bf47cd8b7f2da048ed1cb0, e36c3c425129465fb1cd5aa4e068d3ca
I20260925 01:23:32.139134 11690 raft_consensus.cc:2726] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:23:32.142877 11211 catalog_manager.cc:5495] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f51a07e5bf47cd8b7f2da048ed1cb0 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f51a07e5bf47cd8b7f2da048ed1cb0" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:32.184480 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.179466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.6.236.131:38211)
0925 01:23:32.179620 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.179625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.179747 (+ 122us) server_negotiation.cc:407] Connection header received
0925 01:23:32.179909 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.179913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.180008 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.180134 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:32.180405 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.181165 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.182584 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.183029 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.183124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.183515 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:32.184018 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:32.184123 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.184164 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.184227 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:23:32.192323 11488 raft_consensus.cc:1261] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 1 FOLLOWER]: Refusing update from remote peer 29f51a07e5bf47cd8b7f2da048ed1cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:32.192373 11355 raft_consensus.cc:1261] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 1 FOLLOWER]: Refusing update from remote peer 29f51a07e5bf47cd8b7f2da048ed1cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:32.193197 11691 consensus_queue.cc:1035] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36c3c425129465fb1cd5aa4e068d3ca" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:32.193481 11695 consensus_queue.cc:1035] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc09ac528eb24fa4b669b9cea355734a" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 36943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:23:32.199874 11666 tablet.cc:2369] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:32.216536 11211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:32.216773 11211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:32.220149 11211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31a27a58ed414e6e861ec5e6a91207c5: Sending DeleteTablet for 3 replicas of tablet bc02414e6ffc4cad807fe56d2146a6d2
I20260925 01:23:32.221170 11600 tablet_service.cc:1514] Processing DeleteTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:32 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:43002
I20260925 01:23:32.221271 11468 tablet_service.cc:1514] Processing DeleteTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:32 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:59972
I20260925 01:23:32.221774 11335 tablet_service.cc:1514] Processing DeleteTablet for tablet bc02414e6ffc4cad807fe56d2146a6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:32 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60812
I20260925 01:23:32.223764 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11273
I20260925 01:23:32.233359 11710 tablet_replica.cc:330] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: stopping tablet replica
I20260925 01:23:32.234016 11710 raft_consensus.cc:2229] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:32.234644 11710 raft_consensus.cc:2258] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:32.235421 11709 tablet_replica.cc:330] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: stopping tablet replica
I20260925 01:23:32.235858 11709 raft_consensus.cc:2229] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:32.236348 11709 raft_consensus.cc:2258] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:32.237499 11710 ts_tablet_manager.cc:1905] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:32.238390 11709 ts_tablet_manager.cc:1905] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:32.241473 11708 tablet_replica.cc:330] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: stopping tablet replica
I20260925 01:23:32.241920 11708 raft_consensus.cc:2229] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:32.242671 11708 raft_consensus.cc:2258] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:32.245117 11710 ts_tablet_manager.cc:1918] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:32.245277 11709 ts_tablet_manager.cc:1918] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:32.245509 11709 log.cc:1196] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/bc02414e6ffc4cad807fe56d2146a6d2
I20260925 01:23:32.246101 11709 ts_tablet_manager.cc:1939] T bc02414e6ffc4cad807fe56d2146a6d2 P e36c3c425129465fb1cd5aa4e068d3ca: Deleting consensus metadata
I20260925 01:23:32.246915 11708 ts_tablet_manager.cc:1905] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:32.247943 11198 catalog_manager.cc:4841] TS e36c3c425129465fb1cd5aa4e068d3ca (127.6.236.130:42041): tablet bc02414e6ffc4cad807fe56d2146a6d2 (table test-table [id=d297e7c4b3434f2699e647e5cd09abd5]) successfully deleted
I20260925 01:23:32.248457 11710 log.cc:1196] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/bc02414e6ffc4cad807fe56d2146a6d2
I20260925 01:23:32.248988 11710 ts_tablet_manager.cc:1939] T bc02414e6ffc4cad807fe56d2146a6d2 P fc09ac528eb24fa4b669b9cea355734a: Deleting consensus metadata
I20260925 01:23:32.250780 11196 catalog_manager.cc:4841] TS fc09ac528eb24fa4b669b9cea355734a (127.6.236.129:36943): tablet bc02414e6ffc4cad807fe56d2146a6d2 (table test-table [id=d297e7c4b3434f2699e647e5cd09abd5]) successfully deleted
I20260925 01:23:32.253569 11708 ts_tablet_manager.cc:1918] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:32.253716 11708 log.cc:1196] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/bc02414e6ffc4cad807fe56d2146a6d2
I20260925 01:23:32.254137 11708 ts_tablet_manager.cc:1939] T bc02414e6ffc4cad807fe56d2146a6d2 P 29f51a07e5bf47cd8b7f2da048ed1cb0: Deleting consensus metadata
I20260925 01:23:32.255402 11198 catalog_manager.cc:4841] TS 29f51a07e5bf47cd8b7f2da048ed1cb0 (127.6.236.131:38211): tablet bc02414e6ffc4cad807fe56d2146a6d2 (table test-table [id=d297e7c4b3434f2699e647e5cd09abd5]) successfully deleted
Sep 25 01:23:32 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:32.262740 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.225216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.6.236.129:36943)
0925 01:23:32.225395 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.225402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.240672 (+ 15270us) server_negotiation.cc:407] Connection header received
0925 01:23:32.240862 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.240868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.240949 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.241040 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:32.244623 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.245687 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.247459 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.248055 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.248147 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.257326 (+ 9179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:32.257337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:32.257352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:32.257386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:32.260148 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.260915 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.260921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.260923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.261054 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.261404 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.261407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.261409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.261744 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:32.261954 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.262306 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.262470 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:23:32.263329 11395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:32.352427 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11273
I20260925 01:23:32.367554 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11406
Sep 25 01:23:32 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:32.383591 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.368318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.6.236.130:42041)
0925 01:23:32.368501 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.368506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.368526 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:23:32.368749 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.368753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.368832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.368907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:32.370434 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.371142 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.372874 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.373268 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.373357 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.378173 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:32.378188 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:32.378204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:32.378273 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:32.380997 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.381752 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.381758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.381760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.381876 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.382227 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.382230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.382231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.382639 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:32.382870 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.383180 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.383354 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:23:32.384296 11528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:32.470378 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11406
I20260925 01:23:32.484602 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11538
Sep 25 01:23:32 dist-test-slave-s0b2 krb5kdc[11168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:32.499817 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:32.485333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.6.236.131:38211)
0925 01:23:32.485558 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:32.485564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:32.485583 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:32.485704 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:32.485710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:32.485792 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:32.485882 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:32.487146 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.487902 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.489075 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:32.489550 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:32.489664 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:32.494007 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:32.494019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:32.494035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:32.494074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:32.497050 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.497826 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.497836 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.497839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.497986 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:32.498415 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:32.498418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:32.498420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:32.498776 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:32.499016 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:32.499360 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 01:23:32.499544 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:23:32.500526 11660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:32.619102 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11538
I20260925 01:23:32.635164 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11181
I20260925 01:23:32.636291 11242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:32.742516 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11181
2026-09-25T01:23:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3896 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T01:23:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:32Z Disabled control of system clock
I20260925 01:23:32.801404 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44463
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:23:33.105065 11719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:33.105333 11719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:33.105417 11719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:33.112347 11719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:33.112447 11719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:33.112491 11719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:33.112545 11719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:33.112587 11719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:33.121455 11719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:44463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:33.122664 11719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:33.124315 11719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:33.135316 11724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:33.135284 11725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:33.135246 11727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:33.136236 11719 server_base.cc:1034] running on GCE node
I20260925 01:23:33.136992 11719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:33.138321 11719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:33.139554 11719 hybrid_clock.cc:648] HybridClock initialized: now 1790299413139500 us; error 46 us; skew 500 ppm
I20260925 01:23:33.139874 11719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:33.141944 11719 webserver.cc:460] Webserver started at http://127.6.236.190:39999/ using document root <none> and password file <none>
I20260925 01:23:33.142637 11719 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:33.142735 11719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:33.143026 11719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:33.145823 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "d8a7098e5e1841989851c6b39f93ea87"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "c8aaaa98de6eb8f47007de7a8b3a611b"
server_key_iv: "eb3305ddd52016b3b8097ba18c42e92c"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.146616 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "d8a7098e5e1841989851c6b39f93ea87"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "c8aaaa98de6eb8f47007de7a8b3a611b"
server_key_iv: "eb3305ddd52016b3b8097ba18c42e92c"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.151405 11719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 01:23:33.154724 11733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:33.156170 11719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:23:33.156335 11719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "d8a7098e5e1841989851c6b39f93ea87"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "c8aaaa98de6eb8f47007de7a8b3a611b"
server_key_iv: "eb3305ddd52016b3b8097ba18c42e92c"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.156535 11719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:33.162163 11719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:33.162950 11719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:33.163224 11719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:33.178916 11719 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:44463
I20260925 01:23:33.178932 11784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:44463 every 8 connection(s)
I20260925 01:23:33.180177 11719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:33.183904 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:33.185840 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11719
I20260925 01:23:33.186025 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:33.186789 7090 external_mini_cluster.cc:1428] Setting key e28080b2f44492de5a2df450a1104b31
I20260925 01:23:33.196195 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:33.188969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.6.236.190:44463)
0925 01:23:33.190303 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:33.190317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:33.190367 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:23:33.192404 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:33.192433 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:33.192942 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:33.193365 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:33.194409 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:33.194421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:33.194438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:33.194459 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:33.194542 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:33.194564 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:33.194783 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:33.194910 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 01:23:33.195184 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":503,"threads_started":1}
I20260925 01:23:33.197223 11785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: Bootstrap starting.
I20260925 01:23:33.200225 11785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:33.201287 11785 log.cc:824] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:33.203759 11785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: No bootstrap required, opened a new log
I20260925 01:23:33.211277 11785 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } }
I20260925 01:23:33.211596 11785 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:33.211675 11785 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a7098e5e1841989851c6b39f93ea87, State: Initialized, Role: FOLLOWER
I20260925 01:23:33.212198 11785 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } }
I20260925 01:23:33.212399 11785 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:33.212517 11785 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:33.212643 11785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:33.214895 11785 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } }
I20260925 01:23:33.215313 11785 leader_election.cc:302] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a7098e5e1841989851c6b39f93ea87; no voters:
I20260925 01:23:33.215816 11785 leader_election.cc:288] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:33.215938 11790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:33.216549 11790 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [term 1 LEADER]: Becoming Leader. State: Replica: d8a7098e5e1841989851c6b39f93ea87, State: Running, Role: LEADER
I20260925 01:23:33.217087 11790 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } }
I20260925 01:23:33.217726 11785 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:33.219519 11791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a7098e5e1841989851c6b39f93ea87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } } }
I20260925 01:23:33.219487 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a7098e5e1841989851c6b39f93ea87. Latest consensus state: current_term: 1 leader_uuid: "d8a7098e5e1841989851c6b39f93ea87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a7098e5e1841989851c6b39f93ea87" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44463 } } }
I20260925 01:23:33.219867 11791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:33.219871 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:33.221971 11795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:33.225932 11795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:33.234172 11795 catalog_manager.cc:1347] Generated new cluster ID: 76136618492541edb1ab9ede7718882b
I20260925 01:23:33.234293 11795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:33.269703 11795 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:33.270200 11795 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:33.283761 11795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: Generated new TSK 0
I20260925 01:23:33.284747 11795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:23:33.291975 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--builtin_ntp_servers=127.6.236.148:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:23:33.607544 11809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:33.607834 11809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:33.607969 11809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:33.614981 11809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:33.615078 11809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:33.615221 11809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:33.624056 11809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:33.625269 11809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:33.627063 11809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:33.639415 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:33.639478 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:33.639457 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:33.640232 11809 server_base.cc:1034] running on GCE node
I20260925 01:23:33.640725 11809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:33.641539 11809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:33.642748 11809 hybrid_clock.cc:648] HybridClock initialized: now 1790299413642688 us; error 46 us; skew 500 ppm
I20260925 01:23:33.643066 11809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:33.648227 11809 webserver.cc:460] Webserver started at http://127.6.236.129:39533/ using document root <none> and password file <none>
I20260925 01:23:33.649039 11809 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:33.649158 11809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:33.649497 11809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:33.652392 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "6b57f7d737274218bfe5affec0b91e86"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "032c3b5230b71c74b04bc18e42a7fbff"
server_key_iv: "c539f15c338169e0b37055259caaa3a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.653159 11809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "6b57f7d737274218bfe5affec0b91e86"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "032c3b5230b71c74b04bc18e42a7fbff"
server_key_iv: "c539f15c338169e0b37055259caaa3a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.658300 11809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:23:33.661609 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:33.662894 11809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:23:33.663040 11809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "6b57f7d737274218bfe5affec0b91e86"
format_stamp: "Formatted at 2026-09-25 01:23:33 on dist-test-slave-s0b2"
server_key: "032c3b5230b71c74b04bc18e42a7fbff"
server_key_iv: "c539f15c338169e0b37055259caaa3a5"
server_key_version: "encryptionkey@0"
I20260925 01:23:33.663240 11809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:33.668462 11809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:33.669255 11809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:33.669479 11809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:33.670384 11809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:33.671690 11809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:33.671782 11809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:33.671881 11809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:33.671948 11809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:33.700230 11809 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:40605
I20260925 01:23:33.700291 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:40605 every 8 connection(s)
I20260925 01:23:33.701522 11809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:33.709223 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:33.704041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:33141 (local address 127.6.236.190:44463)
0925 01:23:33.704446 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:33.704450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:33.705707 (+ 1257us) server_negotiation.cc:407] Connection header received
0925 01:23:33.707325 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:33.707330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:33.707385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:33.707512 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:33.708658 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:33.708663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:33.708666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:33.708683 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:33.708726 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:33.708738 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:33.708834 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:33.708963 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 01:23:33.709011 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":203,"threads_started":1}
I20260925 01:23:33.710086 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:33.704606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44463 (local address 127.6.236.129:33141)
0925 01:23:33.705480 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:33.705516 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:33.707057 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:33.707665 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:33.707673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:33.707766 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:33.708368 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:33.708370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:33.708412 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:33.708499 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:33.708815 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:33.708822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:33.708826 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:23:33.709030 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 01:23:33.709219 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":235,"threads_started":1}
I20260925 01:23:33.711185 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11809
I20260925 01:23:33.711350 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:33.711501 11936 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44463
I20260925 01:23:33.711834 11936 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:33.712224 7090 external_mini_cluster.cc:1428] Setting key 290611781a9d365e9a61eba4688dd1d5
I20260925 01:23:33.712677 11936 heartbeater.cc:511] Master 127.6.236.190:44463 requested a full tablet report, sending...
I20260925 01:23:33.714700 11749 ts_manager.cc:194] Registered new tserver with Master: 6b57f7d737274218bfe5affec0b91e86 (127.6.236.129:40605)
I20260925 01:23:33.715022 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--builtin_ntp_servers=127.6.236.148:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:23:34.020432 11940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:34.020699 11940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:34.020819 11940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:34.027940 11940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:34.028033 11940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:34.028168 11940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:34.037106 11940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:34.038342 11940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:34.040006 11940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:34.052093 11948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:34.052160 11946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:34.052093 11945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:34.053036 11940 server_base.cc:1034] running on GCE node
I20260925 01:23:34.053561 11940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:34.054416 11940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:34.055615 11940 hybrid_clock.cc:648] HybridClock initialized: now 1790299414055567 us; error 42 us; skew 500 ppm
I20260925 01:23:34.055907 11940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:34.058020 11940 webserver.cc:460] Webserver started at http://127.6.236.130:38333/ using document root <none> and password file <none>
I20260925 01:23:34.058697 11940 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:34.058795 11940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:34.059083 11940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:34.061853 11940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "fa7aee8f62d54bce80f01a90186dcd83"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "595dda59e68c6c549cfa6dc7b35210ee"
server_key_iv: "a8befce44b5ce038195da58563a68d10"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.062620 11940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "fa7aee8f62d54bce80f01a90186dcd83"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "595dda59e68c6c549cfa6dc7b35210ee"
server_key_iv: "a8befce44b5ce038195da58563a68d10"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.067296 11940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 01:23:34.070529 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.071786 11940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:23:34.071929 11940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "fa7aee8f62d54bce80f01a90186dcd83"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "595dda59e68c6c549cfa6dc7b35210ee"
server_key_iv: "a8befce44b5ce038195da58563a68d10"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.072130 11940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:34.077639 11940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:34.078438 11940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:34.078680 11940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:34.079512 11940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:34.080806 11940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:34.080899 11940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.081000 11940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:34.081069 11940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.110456 11940 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:45975
I20260925 01:23:34.110526 12066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:45975 every 8 connection(s)
I20260925 01:23:34.111726 11940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:34.119501 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.114120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:60501 (local address 127.6.236.190:44463)
0925 01:23:34.114326 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.114330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.115659 (+ 1329us) server_negotiation.cc:407] Connection header received
0925 01:23:34.117346 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.117352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.117431 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.117527 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.118781 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.118787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.118791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.118808 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.118838 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.118849 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.118970 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.119204 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.119262 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:23:34.120301 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.114621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44463 (local address 127.6.236.130:60501)
0925 01:23:34.115500 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.115541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.117090 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.117752 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.117760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.117888 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.118500 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.118502 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.118538 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.118621 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.118976 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.118987 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.118996 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.119218 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.119415 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":244,"threads_started":1}
I20260925 01:23:34.121713 12067 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44463
I20260925 01:23:34.121722 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 11940
I20260925 01:23:34.121871 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:34.122035 12067 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:34.122454 7090 external_mini_cluster.cc:1428] Setting key 7377f073cca6467eb6d047ed99783ac4
I20260925 01:23:34.122749 12067 heartbeater.cc:511] Master 127.6.236.190:44463 requested a full tablet report, sending...
I20260925 01:23:34.124119 11750 ts_manager.cc:194] Registered new tserver with Master: fa7aee8f62d54bce80f01a90186dcd83 (127.6.236.130:45975)
I20260925 01:23:34.125352 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--builtin_ntp_servers=127.6.236.148:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:23:34.443833 12070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:34.444115 12070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:34.444238 12070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:34.451613 12070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:34.451706 12070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:34.451846 12070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:34.460835 12070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:34.462105 12070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:34.463845 12070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:34.476286 12075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:34.476135 12076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:34.476372 12078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:34.477370 12070 server_base.cc:1034] running on GCE node
I20260925 01:23:34.477957 12070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:34.478919 12070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:34.480132 12070 hybrid_clock.cc:648] HybridClock initialized: now 1790299414480079 us; error 49 us; skew 500 ppm
I20260925 01:23:34.480434 12070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:34.482627 12070 webserver.cc:460] Webserver started at http://127.6.236.131:33491/ using document root <none> and password file <none>
I20260925 01:23:34.483291 12070 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:34.483392 12070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:34.483677 12070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:34.486434 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "5cd819188dab45dd9e66eaab2cc19457"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "2fa9c353b58897be7a68cdf946047d68"
server_key_iv: "8b0d58da4ce109f10fdd3eba43ff64cb"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.487185 12070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "5cd819188dab45dd9e66eaab2cc19457"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "2fa9c353b58897be7a68cdf946047d68"
server_key_iv: "8b0d58da4ce109f10fdd3eba43ff64cb"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.492249 12070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:23:34.495589 12084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.496872 12070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:34.497027 12070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "5cd819188dab45dd9e66eaab2cc19457"
format_stamp: "Formatted at 2026-09-25 01:23:34 on dist-test-slave-s0b2"
server_key: "2fa9c353b58897be7a68cdf946047d68"
server_key_iv: "8b0d58da4ce109f10fdd3eba43ff64cb"
server_key_version: "encryptionkey@0"
I20260925 01:23:34.497233 12070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:34.502538 12070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:34.503252 12070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:34.503482 12070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:34.504350 12070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:34.505622 12070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:34.505712 12070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.505815 12070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:34.505882 12070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:34.535152 12070 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:34059
I20260925 01:23:34.535200 12196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:34059 every 8 connection(s)
I20260925 01:23:34.536375 12070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:34.542835 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12070
I20260925 01:23:34.542982 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:34.543403 7090 external_mini_cluster.cc:1428] Setting key 0583e9799fa2bd945042e7d36c2e5742
I20260925 01:23:34.544039 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.538707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:55137 (local address 127.6.236.190:44463)
0925 01:23:34.538866 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.538870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.540237 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 01:23:34.541829 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.541833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.541889 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.541970 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.543277 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.543282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.543285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.543302 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.543333 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.543345 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.543459 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.543683 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.543755 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:23:34.544955 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.539189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44463 (local address 127.6.236.131:55137)
0925 01:23:34.540083 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.540125 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.541598 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.542153 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.542161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.542279 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.542958 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.542962 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.543006 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.543106 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.543429 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.543440 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.543447 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.543730 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.543969 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260925 01:23:34.546205 12197 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44463
I20260925 01:23:34.546494 12197 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:34.547053 12197 heartbeater.cc:511] Master 127.6.236.190:44463 requested a full tablet report, sending...
I20260925 01:23:34.548230 11750 ts_manager.cc:194] Registered new tserver with Master: 5cd819188dab45dd9e66eaab2cc19457 (127.6.236.131:34059)
I20260925 01:23:34.558363 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:23:34.564815 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.562601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.6.236.190:44463)
0925 01:23:34.562747 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.562751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.563231 (+ 480us) server_negotiation.cc:407] Connection header received
0925 01:23:34.563364 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.563368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.563421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.563505 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.564366 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.564370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.564374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.564387 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.564415 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.564426 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.564531 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.564565 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.564606 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:23:34.568526 11750 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:34.575199 11750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:34.602761 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.586385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38296)
0925 01:23:34.598463 (+ 12078us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.598481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.598595 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.600208 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.600217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.600319 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.601337 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.601341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.601371 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.601469 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.601973 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.601985 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.601992 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.602171 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.602279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11946,"thread_start_us":15326,"threads_started":1}
I20260925 01:23:34.603530 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.586966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.6.236.129:40605)
0925 01:23:34.598036 (+ 11070us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.598046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.598707 (+ 661us) server_negotiation.cc:407] Connection header received
0925 01:23:34.598794 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.598803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.599092 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.599236 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.601667 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.601676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.601691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.601716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.601775 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.601796 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.601915 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.603129 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.603231 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10893,"thread_start_us":10753,"threads_started":1}
I20260925 01:23:34.605170 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.586407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.6.236.130:45975)
0925 01:23:34.601940 (+ 15533us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.601948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.601972 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:23:34.602045 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.602053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.602298 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.602962 (+ 664us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.604152 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.604157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.604166 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.604177 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.604213 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.604225 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.604689 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.604756 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.604858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":14644,"threads_started":1}
I20260925 01:23:34.605253 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.587313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.6.236.131:34059)
0925 01:23:34.601701 (+ 14388us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.601709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.602192 (+ 483us) server_negotiation.cc:407] Connection header received
0925 01:23:34.602291 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.602299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.602515 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.602678 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.604084 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.604092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.604106 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.604128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.604183 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.604202 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.604368 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.604428 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.604519 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14232,"thread_start_us":14056,"threads_started":1}
I20260925 01:23:34.605742 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.586278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:45975 (local address 127.0.0.1:54550)
0925 01:23:34.597306 (+ 11028us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.597347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.597485 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.602479 (+ 4994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.602484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.602507 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.603219 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.603222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.603237 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.603278 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.604333 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.604337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.604341 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.604465 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.604515 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10761,"thread_start_us":10586,"threads_started":1}
I20260925 01:23:34.606153 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.586769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:34059 (local address 127.0.0.1:36662)
0925 01:23:34.601955 (+ 15186us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.601971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.602072 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.603200 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.603204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.603227 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.603845 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.603848 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.603863 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.603905 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.604316 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.604321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.604326 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.604478 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.604521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15075,"thread_start_us":11422,"threads_started":1}
I20260925 01:23:34.607630 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b9c2121740458fbf20992f882ac3 (DEFAULT_TABLE table=test-table [id=38b12f20829245549921d2777ff2c2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:34.609097 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b9c2121740458fbf20992f882ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:34.609333 12132 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b9c2121740458fbf20992f882ac3 (DEFAULT_TABLE table=test-table [id=38b12f20829245549921d2777ff2c2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:34.609485 12002 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b9c2121740458fbf20992f882ac3 (DEFAULT_TABLE table=test-table [id=38b12f20829245549921d2777ff2c2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:34.610535 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b9c2121740458fbf20992f882ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:34.610827 12002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b9c2121740458fbf20992f882ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:34.621598 12215 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Bootstrap starting.
I20260925 01:23:34.624928 12215 tablet_bootstrap.cc:654] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:34.625943 12215 log.cc:824] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:34.626559 12216 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Bootstrap starting.
I20260925 01:23:34.627951 12217 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Bootstrap starting.
I20260925 01:23:34.630149 12216 tablet_bootstrap.cc:654] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:34.630856 12217 tablet_bootstrap.cc:654] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:34.631403 12215 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: No bootstrap required, opened a new log
I20260925 01:23:34.631390 12216 log.cc:824] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:34.631721 12215 ts_tablet_manager.cc:1397] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 01:23:34.631955 12217 log.cc:824] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:34.634482 12216 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: No bootstrap required, opened a new log
I20260925 01:23:34.634709 12216 ts_tablet_manager.cc:1397] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 01:23:34.634752 12217 tablet_bootstrap.cc:492] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: No bootstrap required, opened a new log
I20260925 01:23:34.635002 12217 ts_tablet_manager.cc:1397] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 01:23:34.641664 12216 raft_consensus.cc:348] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.641597 12215 raft_consensus.cc:348] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.641978 12216 raft_consensus.cc:374] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:34.642000 12215 raft_consensus.cc:374] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:34.642088 12215 raft_consensus.cc:729] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd819188dab45dd9e66eaab2cc19457, State: Initialized, Role: FOLLOWER
I20260925 01:23:34.642112 12216 raft_consensus.cc:729] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b57f7d737274218bfe5affec0b91e86, State: Initialized, Role: FOLLOWER
I20260925 01:23:34.642658 12216 consensus_queue.cc:260] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.642750 12215 consensus_queue.cc:260] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.643674 11936 heartbeater.cc:503] Master 127.6.236.190:44463 was elected leader, sending a full tablet report...
I20260925 01:23:34.644121 12216 ts_tablet_manager.cc:1428] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 01:23:34.646481 12217 raft_consensus.cc:348] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.646919 12217 raft_consensus.cc:374] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:34.647022 12217 raft_consensus.cc:729] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7aee8f62d54bce80f01a90186dcd83, State: Initialized, Role: FOLLOWER
I20260925 01:23:34.647716 12217 consensus_queue.cc:260] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.648782 12067 heartbeater.cc:503] Master 127.6.236.190:44463 was elected leader, sending a full tablet report...
I20260925 01:23:34.649758 12197 heartbeater.cc:503] Master 127.6.236.190:44463 was elected leader, sending a full tablet report...
I20260925 01:23:34.650743 12215 ts_tablet_manager.cc:1428] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260925 01:23:34.651078 12217 ts_tablet_manager.cc:1428] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s
I20260925 01:23:34.684540 12223 raft_consensus.cc:482] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:34.684810 12223 raft_consensus.cc:504] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.686515 12223 leader_election.cc:288] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b57f7d737274218bfe5affec0b91e86 (127.6.236.129:40605), 5cd819188dab45dd9e66eaab2cc19457 (127.6.236.131:34059)
I20260925 01:23:34.689973 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.687003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:33513 (local address 127.6.236.131:34059)
0925 01:23:34.687158 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.687163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.687844 (+ 681us) server_negotiation.cc:407] Connection header received
0925 01:23:34.688027 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.688031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.688084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.688168 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.689310 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.689317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.689322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.689343 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.689383 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.689398 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.689526 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.689601 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.689660 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:23:34.690044 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.686947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:40969 (local address 127.6.236.129:40605)
0925 01:23:34.687155 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.687161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.687845 (+ 684us) server_negotiation.cc:407] Connection header received
0925 01:23:34.687986 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.687992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.688061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.688169 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.689310 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.689317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.689322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.689343 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.689383 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.689399 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.689552 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.689601 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.689660 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:23:34.690266 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.686924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:34059 (local address 127.6.236.130:33513)
0925 01:23:34.687740 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.687753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.687909 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.688199 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.688205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.688232 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.689111 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.689113 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.689125 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.689152 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.689492 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.689497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.689502 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.689659 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.689719 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"spinlock_wait_cycles":37120,"thread_start_us":407,"threads_started":1}
I20260925 01:23:34.690335 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.686731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.6.236.130:40969)
0925 01:23:34.687721 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.687746 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:34.687870 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:34.688200 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:34.688205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:34.688232 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:34.689050 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:23:34.689054 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:34.689077 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:23:34.689139 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:34.689492 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:34.689510 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:34.689514 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:23:34.689652 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:23:34.689732 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":24,"thread_start_us":614,"threads_started":1}
I20260925 01:23:34.691247 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d3b9c2121740458fbf20992f882ac3" candidate_uuid: "fa7aee8f62d54bce80f01a90186dcd83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b57f7d737274218bfe5affec0b91e86" is_pre_election: true
I20260925 01:23:34.691243 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d3b9c2121740458fbf20992f882ac3" candidate_uuid: "fa7aee8f62d54bce80f01a90186dcd83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd819188dab45dd9e66eaab2cc19457" is_pre_election: true
I20260925 01:23:34.691713 11891 raft_consensus.cc:2454] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7aee8f62d54bce80f01a90186dcd83 in term 0.
I20260925 01:23:34.691716 12152 raft_consensus.cc:2454] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7aee8f62d54bce80f01a90186dcd83 in term 0.
I20260925 01:23:34.692329 11956 leader_election.cc:302] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd819188dab45dd9e66eaab2cc19457, fa7aee8f62d54bce80f01a90186dcd83; no voters:
I20260925 01:23:34.692674 12223 raft_consensus.cc:2781] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:34.692793 12223 raft_consensus.cc:482] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:34.692890 12223 raft_consensus.cc:3037] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:34.695128 12223 raft_consensus.cc:504] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
I20260925 01:23:34.695744 12223 leader_election.cc:288] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [CANDIDATE]: Term 1 election: Requested vote from peers 6b57f7d737274218bfe5affec0b91e86 (127.6.236.129:40605), 5cd819188dab45dd9e66eaab2cc19457 (127.6.236.131:34059)
I20260925 01:23:34.696174 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d3b9c2121740458fbf20992f882ac3" candidate_uuid: "fa7aee8f62d54bce80f01a90186dcd83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b57f7d737274218bfe5affec0b91e86"
I20260925 01:23:34.696269 12152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d3b9c2121740458fbf20992f882ac3" candidate_uuid: "fa7aee8f62d54bce80f01a90186dcd83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd819188dab45dd9e66eaab2cc19457"
I20260925 01:23:34.696395 11891 raft_consensus.cc:3037] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:34.696461 12152 raft_consensus.cc:3037] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:34.699648 12152 raft_consensus.cc:2454] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7aee8f62d54bce80f01a90186dcd83 in term 1.
I20260925 01:23:34.699654 11891 raft_consensus.cc:2454] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7aee8f62d54bce80f01a90186dcd83 in term 1.
I20260925 01:23:34.700079 11956 leader_election.cc:302] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd819188dab45dd9e66eaab2cc19457, fa7aee8f62d54bce80f01a90186dcd83; no voters:
I20260925 01:23:34.700453 12223 raft_consensus.cc:2781] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:34.700872 12223 raft_consensus.cc:686] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 1 LEADER]: Becoming Leader. State: Replica: fa7aee8f62d54bce80f01a90186dcd83, State: Running, Role: LEADER
I20260925 01:23:34.701365 12223 consensus_queue.cc:237] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } }
W20260925 01:23:34.703629 11937 tablet.cc:2369] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:34.706426 11750 catalog_manager.cc:5495] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa7aee8f62d54bce80f01a90186dcd83 (127.6.236.130). New cstate: current_term: 1 leader_uuid: "fa7aee8f62d54bce80f01a90186dcd83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7aee8f62d54bce80f01a90186dcd83" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 45975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:34.730990 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.729151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.6.236.130:45975)
0925 01:23:34.729299 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.729304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.729393 (+ 89us) server_negotiation.cc:407] Connection header received
0925 01:23:34.729579 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.729583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.729663 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.729744 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.730487 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.730508 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.730512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.730529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.730564 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.730576 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.730674 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.730758 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.730805 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:23:34.739066 12152 raft_consensus.cc:1261] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 1 FOLLOWER]: Refusing update from remote peer fa7aee8f62d54bce80f01a90186dcd83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:34.739070 11891 raft_consensus.cc:1261] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 1 FOLLOWER]: Refusing update from remote peer fa7aee8f62d54bce80f01a90186dcd83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:34.739975 12226 consensus_queue.cc:1035] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b57f7d737274218bfe5affec0b91e86" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:34.740293 12223 consensus_queue.cc:1035] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cd819188dab45dd9e66eaab2cc19457" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:34.751384 12229 mvcc.cc:205] Tried to move back new op lower bound from 7333066402761535488 to 7333066402620968960. Current Snapshot: MvccSnapshot[applied={T|T < 7333066402761535488}]
I20260925 01:23:34.754657 12230 mvcc.cc:205] Tried to move back new op lower bound from 7333066402761535488 to 7333066402620968960. Current Snapshot: MvccSnapshot[applied={T|T < 7333066402761535488}]
I20260925 01:23:34.769379 11749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:34.769626 11749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:34.773088 11749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8a7098e5e1841989851c6b39f93ea87: Sending DeleteTablet for 3 replicas of tablet 80d3b9c2121740458fbf20992f882ac3
I20260925 01:23:34.774083 12002 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d3b9c2121740458fbf20992f882ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:34 UTC) from {username='slave'} at 127.0.0.1:54550
I20260925 01:23:34.774169 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d3b9c2121740458fbf20992f882ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:34 UTC) from {username='slave'} at 127.0.0.1:38296
I20260925 01:23:34.774214 12132 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d3b9c2121740458fbf20992f882ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:34 UTC) from {username='slave'} at 127.0.0.1:36662
I20260925 01:23:34.777293 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11809
I20260925 01:23:34.786592 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.778635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.6.236.129:40605)
0925 01:23:34.778811 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.778817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.782859 (+ 4042us) server_negotiation.cc:407] Connection header received
0925 01:23:34.783747 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.783754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.783873 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.783995 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.785604 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.785611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.785616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.785638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.785677 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.785693 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.785785 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.786166 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.786233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:23:34.787338 12240 tablet_replica.cc:330] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: stopping tablet replica
I20260925 01:23:34.787817 11931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:34.788726 12240 raft_consensus.cc:2229] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:34.789235 12240 raft_consensus.cc:2258] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:34.791479 12239 tablet_replica.cc:330] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: stopping tablet replica
I20260925 01:23:34.791510 12240 ts_tablet_manager.cc:1905] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:34.791929 12239 raft_consensus.cc:2229] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:34.792598 12239 raft_consensus.cc:2258] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:34.794673 12239 ts_tablet_manager.cc:1905] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:34.798470 12240 ts_tablet_manager.cc:1918] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:34.798697 12240 log.cc:1196] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/80d3b9c2121740458fbf20992f882ac3
I20260925 01:23:34.799263 12240 ts_tablet_manager.cc:1939] T 80d3b9c2121740458fbf20992f882ac3 P 5cd819188dab45dd9e66eaab2cc19457: Deleting consensus metadata
I20260925 01:23:34.801364 11735 catalog_manager.cc:4841] TS 5cd819188dab45dd9e66eaab2cc19457 (127.6.236.131:34059): tablet 80d3b9c2121740458fbf20992f882ac3 (table test-table [id=38b12f20829245549921d2777ff2c2f5]) successfully deleted
I20260925 01:23:34.802641 12239 ts_tablet_manager.cc:1918] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:34.802793 12239 log.cc:1196] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/80d3b9c2121740458fbf20992f882ac3
I20260925 01:23:34.803303 12239 ts_tablet_manager.cc:1939] T 80d3b9c2121740458fbf20992f882ac3 P fa7aee8f62d54bce80f01a90186dcd83: Deleting consensus metadata
I20260925 01:23:34.805027 11734 catalog_manager.cc:4841] TS fa7aee8f62d54bce80f01a90186dcd83 (127.6.236.130:45975): tablet 80d3b9c2121740458fbf20992f882ac3 (table test-table [id=38b12f20829245549921d2777ff2c2f5]) successfully deleted
I20260925 01:23:34.882025 12241 tablet_replica.cc:330] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86: stopping tablet replica
I20260925 01:23:34.882587 12241 raft_consensus.cc:2229] T 80d3b9c2121740458fbf20992f882ac3 P 6b57f7d737274218bfe5affec0b91e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:34.882725 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11809
I20260925 01:23:34.897727 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11940
W20260925 01:23:34.897819 11737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.236.129:40605 (error 108)
I20260925 01:23:34.899755 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.899295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38304)
0925 01:23:34.899448 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.899590 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 01:23:34.900069 11737 catalog_manager.cc:4584] TS 6b57f7d737274218bfe5affec0b91e86 (127.6.236.129:40605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80d3b9c2121740458fbf20992f882ac3: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
I20260925 01:23:34.901103 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.898917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.6.236.130:45975)
0925 01:23:34.899103 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:34.899109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:34.899128 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:34.899228 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:34.899234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:34.899300 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:34.899401 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:34.900450 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:34.900457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:34.900461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:34.900478 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:34.900509 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:34.900521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:34.900642 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:34.900766 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 01:23:34.900827 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:23:34.901726 12062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:34.953719 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.953228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38320)
0925 01:23:34.953448 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.953545 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 01:23:34.954749 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:34.954397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38324)
0925 01:23:34.954544 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:34.954620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 01:23:35.008164 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11940
I20260925 01:23:35.015345 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.014837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38334)
0925 01:23:35.015066 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:35.015178 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 01:23:35.016490 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.016121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38344)
0925 01:23:35.016284 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:35.016357 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:23:35.023808 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12070
I20260925 01:23:35.026422 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.024465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.6.236.131:34059)
0925 01:23:35.024638 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:35.024643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:35.024663 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:23:35.024836 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:35.024841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:35.024919 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:35.025000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:35.025773 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:35.025779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:35.025784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:35.025801 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:35.025830 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:35.025841 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:35.025949 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:35.026082 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 01:23:35.026141 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:23:35.027015 12192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:35.097082 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.096573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38356)
0925 01:23:35.096806 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:35.096907 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 01:23:35.098114 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.097722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40605 (local address 127.0.0.1:38366)
0925 01:23:35.097918 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:35.098001 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 01:23:35.112473 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12070
I20260925 01:23:35.126111 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 11719
I20260925 01:23:35.127206 11780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:35.231379 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 11719
2026-09-25T01:23:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2487 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 01:23:35.268090 7090 mini_oidc.cc:152] Starting JWKS server
I20260925 01:23:35.270599 7090 webserver.cc:460] Webserver started at https://127.0.0.1:45157/ using document root <none> and password file <none>
I20260925 01:23:35.270916 7090 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:23:35.271538 7090 webserver.cc:460] Webserver started at http://127.0.0.1:43125/ using document root <none> and password file <none>
2026-09-25T01:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:35Z Disabled control of system clock
I20260925 01:23:35.295480 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44399
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:44399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45157/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/testchainca.cert with env {}
W20260925 01:23:35.600909 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:35.601187 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:35.601279 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:35.608322 12253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:35.608419 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:35.608455 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:35.608516 12253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:35.608560 12253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:23:35.608628 12253 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:23:35.608667 12253 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45157/jwks.json
I20260925 01:23:35.618681 12253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:44399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45157/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:35.619900 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:35.621096 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:35.632176 12258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:35.632215 12261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:35.632241 12259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:35.632943 12253 server_base.cc:1034] running on GCE node
I20260925 01:23:35.633636 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:35.634936 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:35.636126 12253 hybrid_clock.cc:648] HybridClock initialized: now 1790299415636081 us; error 34 us; skew 500 ppm
I20260925 01:23:35.636445 12253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:35.638566 12253 webserver.cc:460] Webserver started at http://127.6.236.190:36175/ using document root <none> and password file <none>
I20260925 01:23:35.639209 12253 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:35.639328 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:35.639611 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:35.642371 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "45552cd1c71541439b81f02f04c58ead"
format_stamp: "Formatted at 2026-09-25 01:23:35 on dist-test-slave-s0b2"
server_key: "f1df29bc9ffbe8b17d754ca3fd4f2916"
server_key_iv: "d109f90ab6b15d3a81d92775b16cc691"
server_key_version: "encryptionkey@0"
I20260925 01:23:35.643110 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "45552cd1c71541439b81f02f04c58ead"
format_stamp: "Formatted at 2026-09-25 01:23:35 on dist-test-slave-s0b2"
server_key: "f1df29bc9ffbe8b17d754ca3fd4f2916"
server_key_iv: "d109f90ab6b15d3a81d92775b16cc691"
server_key_version: "encryptionkey@0"
I20260925 01:23:35.647866 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:23:35.651396 12267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:35.652621 12253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:35.652776 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "45552cd1c71541439b81f02f04c58ead"
format_stamp: "Formatted at 2026-09-25 01:23:35 on dist-test-slave-s0b2"
server_key: "f1df29bc9ffbe8b17d754ca3fd4f2916"
server_key_iv: "d109f90ab6b15d3a81d92775b16cc691"
server_key_version: "encryptionkey@0"
I20260925 01:23:35.652963 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:35.664790 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:35.665500 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:35.665764 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:35.681291 12253 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:44399
I20260925 01:23:35.681311 12318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:44399 every 8 connection(s)
I20260925 01:23:35.682556 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:35.686076 12319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:35.690450 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12253
I20260925 01:23:35.690667 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:35.691434 7090 external_mini_cluster.cc:1428] Setting key dbf50396b5d1c29b575f6689d765033c
I20260925 01:23:35.697126 12319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead: Bootstrap starting.
I20260925 01:23:35.700598 12319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:35.701676 12319 log.cc:824] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:35.704591 12319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead: No bootstrap required, opened a new log
I20260925 01:23:35.705178 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.693218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.6.236.190:44399)
0925 01:23:35.694107 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:35.694118 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:35.694162 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:23:35.695937 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:35.695959 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:35.696402 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:35.696833 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:35.697519 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.699033 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.700907 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.701605 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.701743 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:35.702993 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:35.703003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:35.703017 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:35.703532 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:35.703609 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:35.703630 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:35.703728 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:35.703959 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 01:23:35.704245 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":311,"threads_started":1}
I20260925 01:23:35.709043 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:23:35.712926 12319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } }
I20260925 01:23:35.713248 12319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:35.713336 12319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45552cd1c71541439b81f02f04c58ead, State: Initialized, Role: FOLLOWER
I20260925 01:23:35.713840 12319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } }
I20260925 01:23:35.714036 12319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:35.714161 12319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:35.714326 12319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:35.716363 12319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } }
I20260925 01:23:35.716805 12319 leader_election.cc:302] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45552cd1c71541439b81f02f04c58ead; no voters:
I20260925 01:23:35.717494 12319 leader_election.cc:288] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:35.717643 12324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:35.718286 12324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [term 1 LEADER]: Becoming Leader. State: Replica: 45552cd1c71541439b81f02f04c58ead, State: Running, Role: LEADER
I20260925 01:23:35.718739 12324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } }
I20260925 01:23:35.719295 12319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:35.721382 12326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45552cd1c71541439b81f02f04c58ead. Latest consensus state: current_term: 1 leader_uuid: "45552cd1c71541439b81f02f04c58ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } } }
I20260925 01:23:35.721295 12325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45552cd1c71541439b81f02f04c58ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45552cd1c71541439b81f02f04c58ead" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44399 } } }
I20260925 01:23:35.721747 12326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:35.721747 12325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:35.723536 12332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:35.728436 12332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:35.735802 12332 catalog_manager.cc:1347] Generated new cluster ID: feb82ab838bd4a00a86756d91724a268
I20260925 01:23:35.735908 12332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:35.768474 12332 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:35.769918 12332 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:35.781582 12332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45552cd1c71541439b81f02f04c58ead: Generated new TSK 0
I20260925 01:23:35.782542 12332 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:23:35.825367 12349 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 01:23:35.825505 12349 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:23:35.826807 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:35.805415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.6.236.190:44399)
0925 01:23:35.805613 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:35.805619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:35.805965 (+ 346us) server_negotiation.cc:407] Connection header received
0925 01:23:35.806261 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:35.806267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:35.806358 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:35.806538 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:23:35.806954 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.808095 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.809439 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.810166 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.810318 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:35.810806 (+ 488us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:23:35.826053 (+ 15247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:23:35.826454 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 01:23:35.827016 12322 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 01:23:35.828212 12348 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:35.805296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44399 (local address 127.0.0.1:39142)
0925 01:23:35.805874 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:35.805889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:35.806033 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:35.806522 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:35.806527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:35.806538 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:23:35.806825 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:35.806833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.808216 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.808219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:35.809310 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:35.809321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:35.810345 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:35.810350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:35.810478 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:35.810542 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:23:35.826357 (+ 15815us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:23:35.826708 (+ 351us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.236.190:44399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":281,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.236.190:44399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45157/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 01:23:35.830153 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12253
I20260925 01:23:35.831214 12314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:35.926354 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12253
2026-09-25T01:23:35Z chronyd exiting
I20260925 01:23:35.973773 7090 test_util.cc:175] -----------------------------------------------
I20260925 01:23:35.973908 7090 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299375376939-7090-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 01:23:35.977682 7090 mini_oidc.cc:152] Starting JWKS server
I20260925 01:23:35.979418 7090 webserver.cc:460] Webserver started at https://127.0.0.1:36073/ using document root <none> and password file <none>
I20260925 01:23:35.979614 7090 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:23:35.980185 7090 webserver.cc:460] Webserver started at http://127.0.0.1:42041/ using document root <none> and password file <none>
2026-09-25T01:23:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:35Z Disabled control of system clock
I20260925 01:23:36.003319 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46591
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46591
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36073/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/testchainca.cert with env {}
W20260925 01:23:36.303298 12356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:36.303586 12356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:36.303674 12356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:36.310752 12356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:36.310850 12356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:36.310886 12356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:36.310940 12356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:36.310982 12356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:23:36.311039 12356 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:23:36.311077 12356 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36073/jwks.json
I20260925 01:23:36.320641 12356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36073/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:36.321799 12356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:36.323020 12356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:36.333884 12362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:36.333906 12361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:36.333843 12364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:36.334610 12356 server_base.cc:1034] running on GCE node
I20260925 01:23:36.335330 12356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:36.336632 12356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:36.337831 12356 hybrid_clock.cc:648] HybridClock initialized: now 1790299416337776 us; error 47 us; skew 500 ppm
I20260925 01:23:36.338155 12356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:36.340296 12356 webserver.cc:460] Webserver started at http://127.6.236.190:43437/ using document root <none> and password file <none>
I20260925 01:23:36.340936 12356 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:36.341051 12356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:36.341342 12356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:36.344094 12356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "ab7bb44f3cfa4e07b197602db1277ea5"
format_stamp: "Formatted at 2026-09-25 01:23:36 on dist-test-slave-s0b2"
server_key: "aa07162fb6a2ded62fcee2d79c933e85"
server_key_iv: "bce9aa0e297b285ac2d6ecd7cd6892bc"
server_key_version: "encryptionkey@0"
I20260925 01:23:36.344826 12356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "ab7bb44f3cfa4e07b197602db1277ea5"
format_stamp: "Formatted at 2026-09-25 01:23:36 on dist-test-slave-s0b2"
server_key: "aa07162fb6a2ded62fcee2d79c933e85"
server_key_iv: "bce9aa0e297b285ac2d6ecd7cd6892bc"
server_key_version: "encryptionkey@0"
I20260925 01:23:36.349520 12356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:23:36.352804 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:36.354374 12356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:36.354538 12356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "ab7bb44f3cfa4e07b197602db1277ea5"
format_stamp: "Formatted at 2026-09-25 01:23:36 on dist-test-slave-s0b2"
server_key: "aa07162fb6a2ded62fcee2d79c933e85"
server_key_iv: "bce9aa0e297b285ac2d6ecd7cd6892bc"
server_key_version: "encryptionkey@0"
I20260925 01:23:36.354743 12356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:36.386214 12356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:36.387110 12356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:36.387389 12356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:36.403738 12356 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46591
I20260925 01:23:36.403756 12421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46591 every 8 connection(s)
I20260925 01:23:36.405023 12356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:36.408175 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12356
I20260925 01:23:36.408337 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:36.408676 12422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:36.409042 7090 external_mini_cluster.cc:1428] Setting key 802d3c059c88f4fc05e4c8fdb6b914af
I20260925 01:23:36.423312 12422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5: Bootstrap starting.
I20260925 01:23:36.423372 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:36.410875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.6.236.190:46591)
0925 01:23:36.411993 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:36.412008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:36.412058 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:23:36.413602 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:36.413629 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:36.414205 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:36.414663 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:36.415183 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.417122 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.418373 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.419378 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.419580 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:36.420642 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:36.420652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:36.420670 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:36.421357 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:36.421465 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:36.421491 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:36.421602 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:36.421803 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 01:23:36.422123 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":338,"threads_started":1}
I20260925 01:23:36.426496 12422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:36.427022 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:23:36.427546 12422 log.cc:824] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:36.430267 12422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5: No bootstrap required, opened a new log
I20260925 01:23:36.437202 12422 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } }
I20260925 01:23:36.437537 12422 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:36.437631 12422 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7bb44f3cfa4e07b197602db1277ea5, State: Initialized, Role: FOLLOWER
I20260925 01:23:36.438128 12422 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } }
I20260925 01:23:36.438367 12422 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:36.438488 12422 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:36.438644 12422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:36.440675 12422 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } }
I20260925 01:23:36.441136 12422 leader_election.cc:302] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab7bb44f3cfa4e07b197602db1277ea5; no voters:
I20260925 01:23:36.441648 12422 leader_election.cc:288] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:36.441768 12427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:36.442409 12427 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [term 1 LEADER]: Becoming Leader. State: Replica: ab7bb44f3cfa4e07b197602db1277ea5, State: Running, Role: LEADER
I20260925 01:23:36.442917 12427 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } }
I20260925 01:23:36.443491 12422 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:36.445501 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7bb44f3cfa4e07b197602db1277ea5. Latest consensus state: current_term: 1 leader_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } } }
I20260925 01:23:36.445930 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:36.446610 12428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7bb44f3cfa4e07b197602db1277ea5" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46591 } } }
I20260925 01:23:36.448148 12428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:36.449498 12436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:36.454465 12436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:36.463151 12436 catalog_manager.cc:1347] Generated new cluster ID: 0d92dfb0c0084b58aec054e031651a87
I20260925 01:23:36.463253 12436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:36.480307 12436 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:36.481546 12436 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:36.489352 12436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab7bb44f3cfa4e07b197602db1277ea5: Generated new TSK 0
I20260925 01:23:36.490108 12436 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:23:36.525009 12452 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 01:23:36.525116 12452 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 01:23:36.526350 12451 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:36.506172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46591 (local address 127.0.0.1:59748)
0925 01:23:36.506839 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:36.506854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:36.506996 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:36.507421 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:36.507424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:36.507433 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:23:36.507699 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:36.507707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.509101 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.509105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:36.510145 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:36.510156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.511125 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.511130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:36.511266 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:36.511299 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:23:36.525834 (+ 14535us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:23:36.526030 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.236.190:46591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":407,"threads_started":1}
I20260925 01:23:36.526337 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:36.506331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.6.236.190:46591)
0925 01:23:36.506528 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:36.506535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:36.506931 (+ 396us) server_negotiation.cc:407] Connection header received
0925 01:23:36.507146 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:36.507155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:36.507254 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:36.507407 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:23:36.507850 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.508959 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.510300 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:36.510955 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:36.511085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:36.511548 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:23:36.525637 (+ 14089us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:23:36.526000 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 01:23:36.526459 12425 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59748: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36073/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 01:23:36.527992 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12356
I20260925 01:23:36.528977 12417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:36.619746 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12356
2026-09-25T01:23:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 01:23:36.671449 7090 mini_oidc.cc:152] Starting JWKS server
I20260925 01:23:36.673226 7090 webserver.cc:460] Webserver started at https://127.0.0.1:37789/ using document root <none> and password file <none>
I20260925 01:23:36.673426 7090 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:23:36.673954 7090 webserver.cc:460] Webserver started at http://127.0.0.1:41395/ using document root <none> and password file <none>
2026-09-25T01:23:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:36Z Disabled control of system clock
I20260925 01:23:36.698004 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35769
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:35769
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:23:37.016791 12459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:37.017066 12459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:37.017148 12459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:37.024439 12459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:37.024550 12459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:37.024596 12459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:37.024652 12459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:37.024693 12459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:23:37.024752 12459 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:23:37.024791 12459 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37789/jwks.json
I20260925 01:23:37.033968 12459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:35769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:37.035207 12459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:37.036909 12459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:37.047988 12465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:37.048178 12467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:37.048094 12464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:37.048977 12459 server_base.cc:1034] running on GCE node
I20260925 01:23:37.049688 12459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:37.051044 12459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:37.052235 12459 hybrid_clock.cc:648] HybridClock initialized: now 1790299417052187 us; error 41 us; skew 500 ppm
I20260925 01:23:37.052577 12459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:37.054733 12459 webserver.cc:460] Webserver started at http://127.6.236.190:42467/ using document root <none> and password file <none>
I20260925 01:23:37.055346 12459 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:37.055428 12459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:37.055757 12459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:37.058487 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "2ebf30904d874ccea264cdb666a216ab"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "e41a57ffaa476ac1f88b68316f24c6d9"
server_key_iv: "a76f679792ca2fc7ea1199b707f91810"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.059198 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "2ebf30904d874ccea264cdb666a216ab"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "e41a57ffaa476ac1f88b68316f24c6d9"
server_key_iv: "a76f679792ca2fc7ea1199b707f91810"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.063993 12459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:23:37.067133 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:37.068363 12459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 01:23:37.068547 12459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "2ebf30904d874ccea264cdb666a216ab"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "e41a57ffaa476ac1f88b68316f24c6d9"
server_key_iv: "a76f679792ca2fc7ea1199b707f91810"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.068727 12459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:37.088321 12459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:37.089077 12459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:37.089360 12459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:37.105888 12459 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:35769
I20260925 01:23:37.105921 12524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:35769 every 8 connection(s)
I20260925 01:23:37.107174 12459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:37.110764 12525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:37.114500 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12459
I20260925 01:23:37.114715 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:37.115483 7090 external_mini_cluster.cc:1428] Setting key ce307dd5806d40ebd2a1421b450eecf3
I20260925 01:23:37.121814 12525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab: Bootstrap starting.
I20260925 01:23:37.125097 12525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:37.126232 12525 log.cc:824] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:37.129222 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:37.117292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.6.236.190:35769)
0925 01:23:37.118293 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:37.118304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:37.118344 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:23:37.120463 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:37.120486 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:37.120918 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:37.121228 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:37.121811 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.123139 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.124873 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.125575 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.125705 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.126908 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:37.126919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:37.126941 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:37.127644 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:37.127730 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:37.127750 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:37.127833 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:37.128051 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 01:23:37.128336 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":368,"threads_started":1}
I20260925 01:23:37.129262 12525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab: No bootstrap required, opened a new log
I20260925 01:23:37.132774 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:23:37.137414 12525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } }
I20260925 01:23:37.137738 12525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:37.137820 12525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebf30904d874ccea264cdb666a216ab, State: Initialized, Role: FOLLOWER
I20260925 01:23:37.138365 12525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } }
I20260925 01:23:37.138599 12525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:37.138737 12525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:37.138895 12525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:37.140887 12525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } }
I20260925 01:23:37.141394 12525 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebf30904d874ccea264cdb666a216ab; no voters:
I20260925 01:23:37.141978 12525 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:37.142061 12530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:37.142768 12530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [term 1 LEADER]: Becoming Leader. State: Replica: 2ebf30904d874ccea264cdb666a216ab, State: Running, Role: LEADER
I20260925 01:23:37.143255 12530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } }
I20260925 01:23:37.143901 12525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:37.145821 12531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebf30904d874ccea264cdb666a216ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } } }
I20260925 01:23:37.145834 12532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebf30904d874ccea264cdb666a216ab. Latest consensus state: current_term: 1 leader_uuid: "2ebf30904d874ccea264cdb666a216ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebf30904d874ccea264cdb666a216ab" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35769 } } }
I20260925 01:23:37.146294 12532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:37.146337 12531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:37.149039 12536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:37.153530 12536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:37.161320 12536 catalog_manager.cc:1347] Generated new cluster ID: 5434e7beb19e4ec9b41e84dfd6d47316
I20260925 01:23:37.161446 12536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:37.177102 12536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:37.178844 12536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:37.195103 12536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ebf30904d874ccea264cdb666a216ab: Generated new TSK 0
I20260925 01:23:37.196123 12536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:23:37.227721 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:37.219576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.6.236.190:35769)
0925 01:23:37.219755 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:37.219762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:37.220299 (+ 537us) server_negotiation.cc:407] Connection header received
0925 01:23:37.220529 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:37.220534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:37.220606 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:37.220738 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:23:37.221250 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.221965 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.223954 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.224370 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.224463 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.225034 (+ 571us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:23:37.227120 (+ 2086us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:23:37.227448 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 01:23:37.227834 12528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 01:23:37.229852 12554 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:37.219576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35769 (local address 127.0.0.1:57136)
0925 01:23:37.220197 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:37.220217 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:37.220369 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:37.220766 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:37.220771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:37.220785 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:23:37.221110 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:37.221122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.222099 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.222105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:37.223821 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:37.223837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.224544 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.224551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:37.224744 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.224789 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:23:37.227346 (+ 2557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:23:37.229440 (+ 2094us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.236.190:35769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37789/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1791,"thread_start_us":303,"threads_started":1}
I20260925 01:23:37.231333 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12459
I20260925 01:23:37.232430 12520 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 01:23:37.242267 12555 webserver.cc:533] Webserver: sslize failed:
E20260925 01:23:37.242377 12555 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:23:37.324249 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12459
2026-09-25T01:23:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (704 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T01:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:37Z Disabled control of system clock
I20260925 01:23:37.399507 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43059
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:43059
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/testchainca.cert with env {}
W20260925 01:23:37.703363 12560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:37.703649 12560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:37.703780 12560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:37.710813 12560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:37.710911 12560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:37.710953 12560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:37.711030 12560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:37.711084 12560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:23:37.711163 12560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:23:37.711206 12560 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 01:23:37.720942 12560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:43059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:37.722194 12560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:37.723461 12560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:37.734390 12565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:37.734320 12566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:37.734356 12568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:37.735352 12560 server_base.cc:1034] running on GCE node
I20260925 01:23:37.736034 12560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:37.737301 12560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:37.738525 12560 hybrid_clock.cc:648] HybridClock initialized: now 1790299417738472 us; error 38 us; skew 500 ppm
I20260925 01:23:37.738837 12560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:23:37.740912 12560 webserver.cc:460] Webserver started at http://127.6.236.190:37981/ using document root <none> and password file <none>
I20260925 01:23:37.741564 12560 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:37.741667 12560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:37.741946 12560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:37.744686 12560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "2b9ff01ab42d43429b663d6fdc1b8299"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "b44a4968df97cb200626ff20b9847f94"
server_key_iv: "fcc4862e8b08cbf6821185e9907604ce"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.745415 12560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "2b9ff01ab42d43429b663d6fdc1b8299"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "b44a4968df97cb200626ff20b9847f94"
server_key_iv: "fcc4862e8b08cbf6821185e9907604ce"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.750315 12560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:23:37.753625 12574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:37.754935 12560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:23:37.755086 12560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "2b9ff01ab42d43429b663d6fdc1b8299"
format_stamp: "Formatted at 2026-09-25 01:23:37 on dist-test-slave-s0b2"
server_key: "b44a4968df97cb200626ff20b9847f94"
server_key_iv: "fcc4862e8b08cbf6821185e9907604ce"
server_key_version: "encryptionkey@0"
I20260925 01:23:37.755271 12560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:37.776576 12560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:37.777328 12560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:37.777580 12560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:37.794001 12560 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:43059
I20260925 01:23:37.794020 12625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:43059 every 8 connection(s)
I20260925 01:23:37.795281 12560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:37.798975 12626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:37.803610 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12560
I20260925 01:23:37.803787 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:37.804425 7090 external_mini_cluster.cc:1428] Setting key 9e606342f5bde10a2c0cd50a93ae55be
I20260925 01:23:37.810601 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299: Bootstrap starting.
I20260925 01:23:37.814286 12626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:37.815472 12626 log.cc:824] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:37.818889 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299: No bootstrap required, opened a new log
I20260925 01:23:37.819712 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:37.806344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.6.236.190:43059)
0925 01:23:37.807420 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:37.807436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:37.807493 (+ 57us) server_negotiation.cc:407] Connection header received
0925 01:23:37.809851 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:37.809878 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:37.810535 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:37.810980 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:37.811486 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.813419 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.814747 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.815748 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.815922 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.817049 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:37.817059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:37.817076 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:23:37.817720 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:37.817820 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:23:37.817845 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:37.817972 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:37.818097 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 01:23:37.818436 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":311,"threads_started":1}
I20260925 01:23:37.823151 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:23:37.826555 12626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } }
I20260925 01:23:37.826865 12626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:37.826956 12626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9ff01ab42d43429b663d6fdc1b8299, State: Initialized, Role: FOLLOWER
I20260925 01:23:37.827459 12626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } }
I20260925 01:23:37.827661 12626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:37.827785 12626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:37.827916 12626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:37.830137 12626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } }
I20260925 01:23:37.830653 12626 leader_election.cc:302] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9ff01ab42d43429b663d6fdc1b8299; no voters:
I20260925 01:23:37.831197 12626 leader_election.cc:288] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:37.831331 12631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:37.832018 12631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9ff01ab42d43429b663d6fdc1b8299, State: Running, Role: LEADER
I20260925 01:23:37.832513 12631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } }
I20260925 01:23:37.833048 12626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:37.835055 12633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b9ff01ab42d43429b663d6fdc1b8299. Latest consensus state: current_term: 1 leader_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } } }
I20260925 01:23:37.835387 12633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:37.835712 12632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b9ff01ab42d43429b663d6fdc1b8299" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43059 } } }
I20260925 01:23:37.836041 12632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:37.837240 12637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:37.842629 12637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:37.851565 12637 catalog_manager.cc:1347] Generated new cluster ID: 151767967dcd41e6916b7efce48c1c00
I20260925 01:23:37.851679 12637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:37.865693 12637 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:37.866969 12637 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:37.873607 12637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b9ff01ab42d43429b663d6fdc1b8299: Generated new TSK 0
I20260925 01:23:37.874415 12637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:23:37.927980 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:37.891742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.6.236.190:43059)
0925 01:23:37.891947 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:37.891954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:37.892300 (+ 346us) server_negotiation.cc:407] Connection header received
0925 01:23:37.892502 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:37.892507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:37.892598 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:37.892755 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:23:37.893168 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.894298 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.895732 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.896396 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.896535 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.896979 (+ 444us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:23:37.927222 (+ 30243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:23:37.927657 (+ 435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 01:23:37.928086 12655 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:37.891600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43059 (local address 127.0.0.1:42974)
0925 01:23:37.892207 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:37.892222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:37.892341 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:37.892753 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:37.892756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:37.892765 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:23:37.893025 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:37.893033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.894434 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.894438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:37.895585 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:37.895597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:37.896568 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:37.896572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:37.896692 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:37.896726 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:23:37.927515 (+ 30789us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:23:37.927759 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.236.190:43059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
W20260925 01:23:37.928123 12629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42974: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260925 01:23:37.930006 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12560
I20260925 01:23:37.931247 12621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:38.023756 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12560
2026-09-25T01:23:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (679 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (394 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:38 dist-test-slave-s0b2 krb5kdc[12670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:38 dist-test-slave-s0b2 krb5kdc[12670](info): set up 2 sockets
Sep 25 01:23:38 dist-test-slave-s0b2 krb5kdc[12670](info): commencing operation
krb5kdc: starting...
W20260925 01:23:39.799618 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:39 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:39.985580 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38043
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:38043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:40.292378 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:40.292665 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:40.292759 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:40.299731 12683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:40.299836 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:40.299880 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:40.299944 12683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:40.300004 12683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:40.309990 12683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:38043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:40.311429 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:40.312757 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:40.323835 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:40.323937 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:40.323930 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:40.324810 12683 server_base.cc:1034] running on GCE node
I20260925 01:23:40.325569 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:40.327029 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:40.328230 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790299420328180 us; error 37 us; skew 500 ppm
I20260925 01:23:40.328544 12683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:40 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:40.331465 12683 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:23:40.332798 12683 webserver.cc:460] Webserver started at http://127.6.236.190:33537/ using document root <none> and password file <none>
I20260925 01:23:40.333405 12683 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:40.333500 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:40.333758 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:40.336571 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "75301a5276e44fd9bef2e341e0790f7f"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "09b8d919ac2d458b52e9f30721fca960"
server_key_iv: "63ee532f88d3266a7aff7247eafa7673"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.337275 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "75301a5276e44fd9bef2e341e0790f7f"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "09b8d919ac2d458b52e9f30721fca960"
server_key_iv: "63ee532f88d3266a7aff7247eafa7673"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.342063 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 01:23:40.345429 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:40.346763 12683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:40.346910 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "75301a5276e44fd9bef2e341e0790f7f"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "09b8d919ac2d458b52e9f30721fca960"
server_key_iv: "63ee532f88d3266a7aff7247eafa7673"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.347088 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:40.367655 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:40.371969 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:40.372232 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:40.388479 12683 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:38043
I20260925 01:23:40.388501 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:38043 every 8 connection(s)
I20260925 01:23:40.389763 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:40.393160 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12683
I20260925 01:23:40.393340 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:40.393445 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:40.394016 7090 external_mini_cluster.cc:1428] Setting key 2392f33386076fa178c3d92d0bd6834a
I20260925 01:23:40.405876 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f: Bootstrap starting.
Sep 25 01:23:40 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:40.409209 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:40.410388 12750 log.cc:824] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:40.413341 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f: No bootstrap required, opened a new log
I20260925 01:23:40.416797 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:40.395931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.6.236.190:38043)
0925 01:23:40.396723 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:40.396734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:40.396775 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:23:40.399067 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:40.399091 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:40.399521 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:40.399853 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:40.401554 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:40.402914 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:40.404456 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:40.405145 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:40.405318 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:40.409574 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:40.409583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:40.409598 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:40.409640 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:40.412612 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:40.413393 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:40.413401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:40.413410 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:40.413597 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:40.414035 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:40.414038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:40.414040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:40.414611 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:40.414812 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:40.415353 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 01:23:40.415756 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":341,"threads_started":1}
I20260925 01:23:40.421201 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } }
I20260925 01:23:40.421548 12750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:40.421639 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75301a5276e44fd9bef2e341e0790f7f, State: Initialized, Role: FOLLOWER
I20260925 01:23:40.422168 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } }
I20260925 01:23:40.422432 12750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:40.422566 12750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:40.422710 12750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:40.424746 12750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } }
I20260925 01:23:40.425191 12750 leader_election.cc:302] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75301a5276e44fd9bef2e341e0790f7f; no voters:
I20260925 01:23:40.425760 12750 leader_election.cc:288] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:40.425844 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:40.426608 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [term 1 LEADER]: Becoming Leader. State: Replica: 75301a5276e44fd9bef2e341e0790f7f, State: Running, Role: LEADER
I20260925 01:23:40.427105 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } }
I20260925 01:23:40.427791 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:40.429747 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75301a5276e44fd9bef2e341e0790f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } } }
I20260925 01:23:40.429837 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75301a5276e44fd9bef2e341e0790f7f. Latest consensus state: current_term: 1 leader_uuid: "75301a5276e44fd9bef2e341e0790f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75301a5276e44fd9bef2e341e0790f7f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38043 } } }
I20260925 01:23:40.430131 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:40.430131 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:40.432425 12763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:40.437480 12763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:40.446448 12763 catalog_manager.cc:1347] Generated new cluster ID: 84029c96615b42688eef85fd9e3fa44e
I20260925 01:23:40.446555 12763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:40.467295 12763 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:40.468650 12763 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:40.475478 12763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75301a5276e44fd9bef2e341e0790f7f: Generated new TSK 0
I20260925 01:23:40.476387 12763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:23:40.537582 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:40.881708 12776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:40.881973 12776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:40.882090 12776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:40.889179 12776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:40.889276 12776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:40.889420 12776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:40.899506 12776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:40.900725 12776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:40.902030 12776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:40.914291 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:40.914397 12782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:40.914426 12784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:40.915196 12776 server_base.cc:1034] running on GCE node
I20260925 01:23:40.915725 12776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:40.916648 12776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:40.917837 12776 hybrid_clock.cc:648] HybridClock initialized: now 1790299420917787 us; error 45 us; skew 500 ppm
I20260925 01:23:40.918141 12776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:40 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:40.921029 12776 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:23:40.922418 12776 webserver.cc:460] Webserver started at http://127.6.236.129:37955/ using document root <none> and password file <none>
I20260925 01:23:40.923070 12776 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:40.923177 12776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:40.923461 12776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:40.926340 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "309105c732174bd98bf81b634243b059"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "79a9a0bfdb2ea0f76771fd4ef787da61"
server_key_iv: "aec26fd18b509de020a07b9a2a262162"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.927091 12776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "309105c732174bd98bf81b634243b059"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "79a9a0bfdb2ea0f76771fd4ef787da61"
server_key_iv: "aec26fd18b509de020a07b9a2a262162"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.932188 12776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 01:23:40.935448 12791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:40.936758 12776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:40.936906 12776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "309105c732174bd98bf81b634243b059"
format_stamp: "Formatted at 2026-09-25 01:23:40 on dist-test-slave-s0b2"
server_key: "79a9a0bfdb2ea0f76771fd4ef787da61"
server_key_iv: "aec26fd18b509de020a07b9a2a262162"
server_key_version: "encryptionkey@0"
I20260925 01:23:40.937094 12776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:40.956453 12776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:40.960189 12776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:40.960446 12776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:40.961357 12776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:40.962670 12776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:40.962747 12776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:40.962872 12776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:40.962924 12776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:40.991966 12776 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:38551
I20260925 01:23:40.992012 12903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:38551 every 8 connection(s)
I20260925 01:23:40.993280 12776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:40.998874 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12776
I20260925 01:23:40.999078 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:40.999732 7090 external_mini_cluster.cc:1428] Setting key 53838a95f1048add4d5bd764ddadf04b
Sep 25 01:23:40 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:41.012495 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:40.995672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:60195 (local address 127.6.236.190:38043)
0925 01:23:40.996113 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:40.996120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:40.997066 (+ 946us) server_negotiation.cc:407] Connection header received
0925 01:23:40.998863 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:40.998870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:40.998955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:40.999078 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:41.000651 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.001541 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.003114 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.003671 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.003790 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.007127 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:41.007134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:41.007139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:41.007176 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:41.009570 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.010386 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.010390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.010392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.010545 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.010966 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.010969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.010971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.011211 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:41.011315 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:41.012089 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 01:23:41.012253 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":203,"threads_started":1}
I20260925 01:23:41.013788 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:40.996216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38043 (local address 127.6.236.129:60195)
0925 01:23:40.996902 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:40.996940 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:40.998586 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:40.999302 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:40.999311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:40.999815 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:41.000498 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.000522 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.001693 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.001697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.002968 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.002980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.003880 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.003884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.004086 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.004699 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:41.004725 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:41.006898 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:41.009702 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.009710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.009726 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.010277 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.010643 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.010647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.010650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.010877 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.011313 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:41.011320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:41.011719 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 01:23:41.012023 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 01:23:41.012507 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260925 01:23:41.015254 12904 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38043
I20260925 01:23:41.015545 12904 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:41.016237 12904 heartbeater.cc:511] Master 127.6.236.190:38043 requested a full tablet report, sending...
I20260925 01:23:41.018127 12715 ts_manager.cc:194] Registered new tserver with Master: 309105c732174bd98bf81b634243b059 (127.6.236.129:38551)
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:23:41.019538 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:60195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:23:41.041734 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:41.349107 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:41.349355 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:41.349452 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:41.356485 12910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:41.356588 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:41.356698 12910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:41.366071 12910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:41.367393 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:41.368647 12910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:41.380710 12915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:41.380793 12916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:41.380828 12918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:41.381538 12910 server_base.cc:1034] running on GCE node
I20260925 01:23:41.382097 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:41.382977 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:41.384166 12910 hybrid_clock.cc:648] HybridClock initialized: now 1790299421384106 us; error 54 us; skew 500 ppm
I20260925 01:23:41.384433 12910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:41.387320 12910 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:23:41.388676 12910 webserver.cc:460] Webserver started at http://127.6.236.130:42947/ using document root <none> and password file <none>
I20260925 01:23:41.389292 12910 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:41.389400 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:41.389673 12910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:41.392421 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "a2798dd83fa94d7ba91a2562defbda23"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "694e20c764936d6857017042a36db9e2"
server_key_iv: "860c9ed19a0693fef4e1dc9c03aa950d"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.393153 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "a2798dd83fa94d7ba91a2562defbda23"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "694e20c764936d6857017042a36db9e2"
server_key_iv: "860c9ed19a0693fef4e1dc9c03aa950d"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.398156 12910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 01:23:41.401418 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.402805 12910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:41.402943 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "a2798dd83fa94d7ba91a2562defbda23"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "694e20c764936d6857017042a36db9e2"
server_key_iv: "860c9ed19a0693fef4e1dc9c03aa950d"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.403122 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:41.419143 12910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:41.423243 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:41.423492 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:41.424440 12910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:41.425711 12910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:41.425778 12910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.425885 12910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:41.425936 12910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.455427 12910 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:39543
I20260925 01:23:41.455468 13037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:39543 every 8 connection(s)
I20260925 01:23:41.456605 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:41.458711 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 12910
I20260925 01:23:41.458863 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:41.459365 7090 external_mini_cluster.cc:1428] Setting key 43640aed4eb947427d2b5a68894793c8
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:41.476483 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:41.458996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:45335 (local address 127.6.236.190:38043)
0925 01:23:41.459204 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:41.459210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:41.460448 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 01:23:41.462263 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:41.462269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:41.462354 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:41.462459 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:41.463902 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.464856 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.466375 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.466862 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.466958 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.470982 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:41.470987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:41.470990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:41.471025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:41.473413 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.474264 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.474270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.474273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.474400 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.474868 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.474873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.474876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.475136 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:41.475243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:41.476080 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 01:23:41.476247 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:41.477941 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:41.459511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38043 (local address 127.6.236.130:45335)
0925 01:23:41.460291 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:41.460336 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:41.461980 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:41.462679 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:41.462687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:41.463195 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:41.463744 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.463761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.465013 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.465017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.466228 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.466263 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.467137 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.467145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.467435 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.468105 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:41.468131 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:41.470762 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:41.473551 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.473561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.473578 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.474146 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.474500 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.474505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.474508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.474746 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.475255 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:41.475263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:41.475702 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 01:23:41.476015 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 01:23:41.476554 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:41.479372 13038 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38043
I20260925 01:23:41.479666 13038 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:41.480408 13038 heartbeater.cc:511] Master 127.6.236.190:38043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
I20260925 01:23:41.481938 12715 ts_manager.cc:194] Registered new tserver with Master: a2798dd83fa94d7ba91a2562defbda23 (127.6.236.130:39543)
I20260925 01:23:41.482971 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:45335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:23:41.503134 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:41.812435 13043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:41.812718 13043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:41.812860 13043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:41.819900 13043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:41.820016 13043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:41.820168 13043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:41.829705 13043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:41.830981 13043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:41.832341 13043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:41.845116 13048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:41.845145 13049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:41.845283 13051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:41.846164 13043 server_base.cc:1034] running on GCE node
I20260925 01:23:41.846757 13043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:41.847610 13043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:41.848820 13043 hybrid_clock.cc:648] HybridClock initialized: now 1790299421848759 us; error 45 us; skew 500 ppm
I20260925 01:23:41.849129 13043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:41.852102 13043 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:23:41.853449 13043 webserver.cc:460] Webserver started at http://127.6.236.131:35629/ using document root <none> and password file <none>
I20260925 01:23:41.854082 13043 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:41.854182 13043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:41.854480 13043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:41.857322 13043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "9a30b786502a42b9904ea1d04a9b891e"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "fd648b91452025051e6cb2c99b4fc24d"
server_key_iv: "353689b9b4919e718f760a1e498f6011"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.858038 13043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "9a30b786502a42b9904ea1d04a9b891e"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "fd648b91452025051e6cb2c99b4fc24d"
server_key_iv: "353689b9b4919e718f760a1e498f6011"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.862970 13043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 01:23:41.866169 13058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.867435 13043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:23:41.867584 13043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "9a30b786502a42b9904ea1d04a9b891e"
format_stamp: "Formatted at 2026-09-25 01:23:41 on dist-test-slave-s0b2"
server_key: "fd648b91452025051e6cb2c99b4fc24d"
server_key_iv: "353689b9b4919e718f760a1e498f6011"
server_key_version: "encryptionkey@0"
I20260925 01:23:41.867759 13043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:41.886785 13043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:41.890527 13043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:41.890779 13043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:41.891685 13043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:41.892966 13043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:41.893044 13043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.893165 13043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:41.893218 13043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:41.922765 13043 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:37207
I20260925 01:23:41.922809 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:37207 every 8 connection(s)
I20260925 01:23:41.924019 13043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:41.934387 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13043
I20260925 01:23:41.934580 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:41.935065 7090 external_mini_cluster.cc:1428] Setting key d74ea1bb6f0a0f2f344698e3b165e867
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:23:41.942906 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:41.926401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:51189 (local address 127.6.236.190:38043)
0925 01:23:41.926608 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:41.926614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:41.927802 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 01:23:41.929583 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:41.929588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:41.929656 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:41.929741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:41.931157 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.931989 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.933477 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.933886 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.933980 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.937587 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:41.937592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:41.937595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:41.937627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:41.940161 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.940870 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.940874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.940876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.940991 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.941412 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.941416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.941418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.941717 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:41.941823 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:41.942479 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 01:23:41.942666 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:23:41.943740 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:41.926948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38043 (local address 127.6.236.131:51189)
0925 01:23:41.927662 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:41.927699 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:41.929320 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:41.929920 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:41.929928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:41.930421 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:41.930981 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.930998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.932127 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.932131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.933356 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:41.933368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.934084 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.934089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:41.934310 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.934934 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:41.934960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:41.937378 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:41.940277 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.940285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.940300 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.940766 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.941111 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:41.941116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:41.941120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:41.941325 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:41.941815 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:41.941821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:41.942125 (+ 304us) client_negotiation.cc:769] Sending connection context
0925 01:23:41.942385 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 01:23:41.942743 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":293,"threads_started":1}
I20260925 01:23:41.944937 13171 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38043
I20260925 01:23:41.945214 13171 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:41.945901 13171 heartbeater.cc:511] Master 127.6.236.190:38043 requested a full tablet report, sending...
I20260925 01:23:41.947259 12715 ts_manager.cc:194] Registered new tserver with Master: 9a30b786502a42b9904ea1d04a9b891e (127.6.236.131:37207)
I20260925 01:23:41.948041 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:51189
I20260925 01:23:41.950377 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/test-admin.keytab.
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:41.975111 7090 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 01:23:41.979780 13178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:41.977621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.6.236.129:38551)
0925 01:23:41.978166 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:41.978175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:41.978206 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:23:41.978304 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:41.978313 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:41.978578 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:41.978730 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:41.979452 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32836: BlockingRecv error: recv got EOF from 127.0.0.1:32836 (error 108)
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
W20260925 01:23:41.979969 7090 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:41.981561 7090 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:41.982827 7090 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:41.997174 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:41.984310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.6.236.129:38551)
0925 01:23:41.984487 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:41.984492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:41.984622 (+ 130us) server_negotiation.cc:407] Connection header received
0925 01:23:41.984740 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:41.984743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:41.984796 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:41.984869 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:41.985749 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.986745 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.987904 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:41.988415 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:41.988503 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:41.991761 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:41.991771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:41.991787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:41.991819 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:41.994582 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.995214 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.995221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.995224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.995346 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:41.995711 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:41.995715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:41.995718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:41.996063 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:41.996336 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:41.996684 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 01:23:41.996925 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:23:42.002193 13178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:42.000739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.6.236.129:38551)
0925 01:23:42.000927 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.000932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.001094 (+ 162us) server_negotiation.cc:407] Connection header received
0925 01:23:42.001215 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.001219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.001268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.001332 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.001991 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32860: BlockingRecv error: recv got EOF from 127.0.0.1:32860 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 01:23:42.002419 7090 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:42.003787 7090 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:23:41 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:42.004940 7090 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:42.017072 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:42.006075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.6.236.129:38551)
0925 01:23:42.006317 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.006322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.006524 (+ 202us) server_negotiation.cc:407] Connection header received
0925 01:23:42.006635 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.006638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.006689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.006759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.007642 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.008332 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.009411 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.009794 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.009885 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:42.012994 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:42.012998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:42.013001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:42.013041 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:42.015212 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.015746 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.015750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.015752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.015857 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.016134 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.016137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.016139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.016365 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:42.016541 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:42.016712 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 01:23:42.016859 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 01:23:42.020699 13178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:42.019372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.6.236.129:38551)
0925 01:23:42.019512 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.019516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.019780 (+ 264us) server_negotiation.cc:407] Connection header received
0925 01:23:42.019882 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.019885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.019942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.020012 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.020498 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32870: BlockingRecv error: recv got EOF from 127.0.0.1:32870 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 01:23:42.021126 7090 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299375376939-7090-0/krb5kdc/krb5cc)
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:42.022219 12904 heartbeater.cc:503] Master 127.6.236.190:38043 was elected leader, sending a full tablet report...
I20260925 01:23:42.022632 7090 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:42.024020 7090 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:23:42.036630 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:42.025130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.6.236.129:38551)
0925 01:23:42.025290 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.025295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.025584 (+ 289us) server_negotiation.cc:407] Connection header received
0925 01:23:42.025723 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.025728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.025781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.025849 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.026822 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.027511 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.028774 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.029191 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.029294 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:42.032489 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:42.032493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:42.032496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:42.032526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:42.034687 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.035206 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.035210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.035212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.035345 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.035613 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.035616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.035618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.035856 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:42.036062 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:42.036246 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 01:23:42.036391 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:23:42.038021 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12776
I20260925 01:23:42.048246 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:42.038690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.6.236.129:38551)
0925 01:23:42.038830 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.038834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.039110 (+ 276us) server_negotiation.cc:407] Connection header received
0925 01:23:42.039219 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.039223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.039287 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.039382 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.040183 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.040845 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.041978 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.042432 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.042528 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:42.044177 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:42.044181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:42.044185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:42.044211 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:42.046358 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.046890 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.046893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.046895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.047013 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.047265 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.047268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.047270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.047490 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:42.047676 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:42.047878 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 01:23:42.048020 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:23:42.048763 12899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:42.130079 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12776
I20260925 01:23:42.143913 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12910
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:23:42.159202 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:42.144848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.6.236.130:39543)
0925 01:23:42.145792 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.145799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.145826 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:23:42.145894 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.145900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.146127 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.146293 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.147254 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.148283 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.149567 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.150213 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.150375 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:42.153984 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:42.153995 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:42.154010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:42.154046 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:42.156676 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.157266 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.157272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.157274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.157388 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.157724 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.157727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.157729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.158049 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:42.158305 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:42.158709 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 01:23:42.158903 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":646,"threads_started":1}
I20260925 01:23:42.160445 13033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:42.239933 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12910
I20260925 01:23:42.252904 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13043
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[12670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:23:42.268167 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:42.253913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.6.236.131:37207)
0925 01:23:42.254513 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:42.254522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:42.254554 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:23:42.254619 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:42.254625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:42.254837 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:42.254988 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:42.256076 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.257092 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.258349 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:42.258914 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:42.259036 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:42.262768 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:42.262779 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:42.262794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:42.262830 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:42.265372 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.266021 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.266028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.266030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.266154 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:42.266688 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:42.266692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:42.266694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:42.267046 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:42.267328 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:42.267633 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 01:23:42.267881 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
I20260925 01:23:42.269367 13166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:42.348781 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13043
I20260925 01:23:42.361994 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 12683
I20260925 01:23:42.363171 12745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:42.463442 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 12683
2026-09-25T01:23:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3660 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[13198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[13198](info): set up 2 sockets
Sep 25 01:23:42 dist-test-slave-s0b2 krb5kdc[13198](info): commencing operation
krb5kdc: starting...
W20260925 01:23:43.467677 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:43 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab.
Entry for principal oryx/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab.
I20260925 01:23:43.651566 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43315
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab
--principal=oryx/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:43.961076 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:43.961345 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:43.961475 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:43.968497 13211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:43.968601 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:43.968676 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:43.968751 13211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:43.968806 13211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:43.978394 13211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab
--principal=oryx/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:43.979655 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:43.980926 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:43.991914 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:43.991846 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:43.991887 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:43.993057 13211 server_base.cc:1034] running on GCE node
I20260925 01:23:43.993808 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:43.995253 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:43.996464 13211 hybrid_clock.cc:648] HybridClock initialized: now 1790299423996393 us; error 61 us; skew 500 ppm
I20260925 01:23:43.996798 13211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:43 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:44.004693 13211 init.cc:377] Logged in from keytab as oryx/127.6.236.190@KRBTEST.COM (short username oryx)
I20260925 01:23:44.006165 13211 webserver.cc:460] Webserver started at http://127.6.236.190:34553/ using document root <none> and password file <none>
I20260925 01:23:44.006889 13211 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:44.007006 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:44.007285 13211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:44.010147 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "38b03a55932f4ed2a155f9c851b60407"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "1c01052c06c0bfb36da9cc20d696bf87"
server_key_iv: "1e9e1eed66b398d189a5634037155a8c"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.010918 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "38b03a55932f4ed2a155f9c851b60407"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "1c01052c06c0bfb36da9cc20d696bf87"
server_key_iv: "1e9e1eed66b398d189a5634037155a8c"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.015810 13211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:23:44.019090 13226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:44.020372 13211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:44.020530 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "38b03a55932f4ed2a155f9c851b60407"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "1c01052c06c0bfb36da9cc20d696bf87"
server_key_iv: "1e9e1eed66b398d189a5634037155a8c"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.020702 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:44.041402 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:44.045185 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:44.045406 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:44.062362 13211 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:43315
I20260925 01:23:44.062425 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:43315 every 8 connection(s)
I20260925 01:23:44.063694 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:44.067340 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:44.068679 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13211
I20260925 01:23:44.068845 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:44.069516 7090 external_mini_cluster.cc:1428] Setting key 362b2f062cea95994783e60afcbc95ad
I20260925 01:23:44.080847 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Bootstrap starting.
Sep 25 01:23:44 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:44.084512 13278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:44.085564 13278 log.cc:824] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:44.088726 13278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: No bootstrap required, opened a new log
I20260925 01:23:44.092131 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:44.071437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.6.236.190:43315)
0925 01:23:44.072190 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:44.072202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:44.072245 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:23:44.073763 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:44.073785 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:44.074165 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:44.074528 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:44.076212 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.077565 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.079362 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.080016 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.080148 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:44.084468 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:44.084476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:44.084491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:44.084537 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:44.087822 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:44.088444 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:44.088453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:44.088471 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:44.088682 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:44.089022 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:44.089027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:44.089030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:44.089733 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:44.090050 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:44.090332 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 01:23:44.090792 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":343,"threads_started":1}
I20260925 01:23:44.097541 13278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } }
I20260925 01:23:44.097939 13278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:44.098052 13278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b03a55932f4ed2a155f9c851b60407, State: Initialized, Role: FOLLOWER
I20260925 01:23:44.098583 13278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } }
I20260925 01:23:44.098784 13278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:44.098905 13278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:44.099059 13278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:44.101063 13278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } }
I20260925 01:23:44.101536 13278 leader_election.cc:302] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b03a55932f4ed2a155f9c851b60407; no voters:
I20260925 01:23:44.102178 13278 leader_election.cc:288] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:44.102275 13283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:44.103045 13283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 LEADER]: Becoming Leader. State: Replica: 38b03a55932f4ed2a155f9c851b60407, State: Running, Role: LEADER
I20260925 01:23:44.103545 13283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } }
I20260925 01:23:44.104193 13278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:44.106007 13284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } }
I20260925 01:23:44.106376 13284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:44.106801 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b03a55932f4ed2a155f9c851b60407. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } }
I20260925 01:23:44.107136 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:44.108163 13289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:44.112360 13289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:44.121198 13289 catalog_manager.cc:1347] Generated new cluster ID: 6a364ac311484adf97fb375bc795e735
I20260925 01:23:44.121305 13289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:44.137528 13289 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:44.138859 13289 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:44.147342 13289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Generated new TSK 0
I20260925 01:23:44.148136 13289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
I20260925 01:23:44.197597 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:44.507050 13304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:44.507311 13304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:44.507532 13304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:44.514565 13304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:44.514659 13304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:44.514801 13304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:44.524214 13304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:44.525447 13304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:44.526729 13304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:44.538947 13309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:44.538969 13312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:44.539026 13310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:44.539760 13304 server_base.cc:1034] running on GCE node
I20260925 01:23:44.540323 13304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:44.541188 13304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:44.542326 13304 hybrid_clock.cc:648] HybridClock initialized: now 1790299424542278 us; error 41 us; skew 500 ppm
I20260925 01:23:44.542631 13304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:44 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299424, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:44.545596 13304 init.cc:377] Logged in from keytab as oryx/127.6.236.129@KRBTEST.COM (short username oryx)
I20260925 01:23:44.547094 13304 webserver.cc:460] Webserver started at http://127.6.236.129:40411/ using document root <none> and password file <none>
I20260925 01:23:44.547775 13304 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:44.547892 13304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:44.548172 13304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:44.551215 13304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "a434b62f519e288e00abb7f4718a7571"
server_key_iv: "915d6d80e8965d262dce022a9399f3ed"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.551959 13304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "a434b62f519e288e00abb7f4718a7571"
server_key_iv: "915d6d80e8965d262dce022a9399f3ed"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.557118 13304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:23:44.560595 13319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:44.561947 13304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 01:23:44.562096 13304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "a434b62f519e288e00abb7f4718a7571"
server_key_iv: "915d6d80e8965d262dce022a9399f3ed"
server_key_version: "encryptionkey@0"
I20260925 01:23:44.562304 13304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:44.584976 13304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:44.589203 13304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:44.589459 13304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:44.590415 13304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:44.591701 13304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:44.591773 13304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:44.591884 13304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:44.591934 13304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:44.619937 13304 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42755
I20260925 01:23:44.619992 13431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42755 every 8 connection(s)
I20260925 01:23:44.621151 13304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:44.626387 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13304
I20260925 01:23:44.626565 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:44.627189 7090 external_mini_cluster.cc:1428] Setting key 8e1e9c057bb402a42a819dde5ba05f5b
Sep 25 01:23:44 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299424, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:44.639815 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:44.623482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:56387 (local address 127.6.236.190:43315)
0925 01:23:44.623908 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:44.623913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:44.624836 (+ 923us) server_negotiation.cc:407] Connection header received
0925 01:23:44.626662 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:44.626668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:44.626752 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:44.626846 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:44.628434 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.629274 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.630815 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.631200 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.631311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:44.634630 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:44.634634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:44.634638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:44.634670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:44.636946 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:44.637741 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:44.637744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:44.637747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:44.637855 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:44.638287 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:44.638292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:44.638294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:44.638552 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:44.638661 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:44.639391 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 01:23:44.639570 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":215,"threads_started":1}
I20260925 01:23:44.641038 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:44.623973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:56387)
0925 01:23:44.624666 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:44.624707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:44.626392 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:44.627095 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:44.627103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:44.627617 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:44.628270 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:44.628287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.629412 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.629416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:44.630686 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:44.630699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:44.631412 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:44.631420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:44.631633 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:44.632241 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:44.632262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:44.634427 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:44.637086 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:44.637094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:44.637109 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:44.637647 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:44.637934 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:44.637938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:44.637941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:44.638160 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:44.638663 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:44.638671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:44.639052 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 01:23:44.639315 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 01:23:44.639779 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":255,"threads_started":1}
I20260925 01:23:44.642561 13432 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:44.642853 13432 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:44.643576 13432 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:23:44.645623 13243 ts_manager.cc:194] Registered new tserver with Master: e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:44.647168 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:56387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
I20260925 01:23:44.671226 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:44.977049 13438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:44.977330 13438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:44.977476 13438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:44.984738 13438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:44.984859 13438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:44.985029 13438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:44.994866 13438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:44.996147 13438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:44.997421 13438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:45.009840 13443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.009775 13446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.009814 13444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:45.010849 13438 server_base.cc:1034] running on GCE node
I20260925 01:23:45.011437 13438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:45.012318 13438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:45.013517 13438 hybrid_clock.cc:648] HybridClock initialized: now 1790299425013462 us; error 46 us; skew 500 ppm
I20260925 01:23:45.013809 13438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:45 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:45.016872 13438 init.cc:377] Logged in from keytab as oryx/127.6.236.130@KRBTEST.COM (short username oryx)
I20260925 01:23:45.018214 13438 webserver.cc:460] Webserver started at http://127.6.236.130:40317/ using document root <none> and password file <none>
I20260925 01:23:45.018898 13438 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:45.019011 13438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:45.019294 13438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:45.022047 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "08e99fa0e0a040878a25b6de75fc6871"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "fbc6a49279bdfb5b91464de79d602689"
server_key_iv: "f3f70c35e3e3f2ee60355c210786ba5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.022814 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "08e99fa0e0a040878a25b6de75fc6871"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "fbc6a49279bdfb5b91464de79d602689"
server_key_iv: "f3f70c35e3e3f2ee60355c210786ba5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.027724 13438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:23:45.031268 13453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.032586 13438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:45.032734 13438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "08e99fa0e0a040878a25b6de75fc6871"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "fbc6a49279bdfb5b91464de79d602689"
server_key_iv: "f3f70c35e3e3f2ee60355c210786ba5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.032914 13438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:45.052573 13438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:45.056681 13438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:45.056936 13438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:45.057865 13438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:45.059176 13438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:45.059249 13438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.059368 13438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:45.059420 13438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.093401 13438 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43397
I20260925 01:23:45.093417 13565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43397 every 8 connection(s)
I20260925 01:23:45.094656 13438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:45.099100 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13438
I20260925 01:23:45.099274 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:45.099830 7090 external_mini_cluster.cc:1428] Setting key d1ec8eb85397d171bb6c67cdb74a0ca3
Sep 25 01:23:45 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:45.115451 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:45.097215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:37473 (local address 127.6.236.190:43315)
0925 01:23:45.097399 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:45.097404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:45.098589 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 01:23:45.100941 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:45.100948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:45.101048 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:45.101152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:45.102971 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.103703 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.105707 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.106127 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.106223 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:45.109875 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:45.109880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:45.109883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:45.109917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:45.112466 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:45.113177 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:45.113181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:45.113183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:45.113312 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:45.113714 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:45.113719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:45.113722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:45.114027 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:45.114157 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:45.114905 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 01:23:45.115133 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:45.116493 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:45.097667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:37473)
0925 01:23:45.098404 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:45.098458 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:45.100654 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:45.101330 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:45.101338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:45.101973 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:45.102800 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:45.102828 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.103861 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.103866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:45.105560 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:45.105574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.106357 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.106364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:45.106643 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:45.107303 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:45.107335 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:45.109662 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:45.112627 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:45.112636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:45.112653 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:45.113060 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:45.113415 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:45.113421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:45.113425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:45.113602 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:45.114142 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:45.114150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:45.114537 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 01:23:45.114768 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 01:23:45.115339 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
I20260925 01:23:45.117823 13566 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:45.118142 13566 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:45.118870 13566 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:23:45.120236 13243 ts_manager.cc:194] Registered new tserver with Master: 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:45.121062 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:37473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
I20260925 01:23:45.145112 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:45.453627 13571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:45.453895 13571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:45.453989 13571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:45.461012 13571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:45.461117 13571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:45.461226 13571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:45.470664 13571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:45.471832 13571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:45.473052 13571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:45.485190 13576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.485211 13579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.485235 13577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:45.486104 13571 server_base.cc:1034] running on GCE node
I20260925 01:23:45.486675 13571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:45.487534 13571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:45.488731 13571 hybrid_clock.cc:648] HybridClock initialized: now 1790299425488675 us; error 49 us; skew 500 ppm
I20260925 01:23:45.489019 13571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:45 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:45.491978 13571 init.cc:377] Logged in from keytab as oryx/127.6.236.131@KRBTEST.COM (short username oryx)
I20260925 01:23:45.493296 13571 webserver.cc:460] Webserver started at http://127.6.236.131:38089/ using document root <none> and password file <none>
I20260925 01:23:45.493929 13571 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:45.494045 13571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:45.494356 13571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:45.497061 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "bcd5bc3f63764e8198a3441bd27b1023"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a4382ac7c81ca9ceb662773a624e9cb"
server_key_iv: "7a7efe9bcee856e7fc4a25e6e4d179ca"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.497766 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "bcd5bc3f63764e8198a3441bd27b1023"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a4382ac7c81ca9ceb662773a624e9cb"
server_key_iv: "7a7efe9bcee856e7fc4a25e6e4d179ca"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.502683 13571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 01:23:45.505863 13586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.507218 13571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:45.507359 13571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "bcd5bc3f63764e8198a3441bd27b1023"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a4382ac7c81ca9ceb662773a624e9cb"
server_key_iv: "7a7efe9bcee856e7fc4a25e6e4d179ca"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.507525 13571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:45.525164 13571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:45.528980 13571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:45.529238 13571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:45.530143 13571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:45.531405 13571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:45.531478 13571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.531603 13571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:45.531656 13571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.560882 13571 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:33891
I20260925 01:23:45.560916 13698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:33891 every 8 connection(s)
I20260925 01:23:45.562103 13571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:45.562981 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13571
I20260925 01:23:45.563169 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:45.563704 7090 external_mini_cluster.cc:1428] Setting key 2069a88656abe0b6c14c0d598c0ec3e1
Sep 25 01:23:45 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:45.581660 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:45.564966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:45771 (local address 127.6.236.190:43315)
0925 01:23:45.565144 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:45.565150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:45.566702 (+ 1552us) server_negotiation.cc:407] Connection header received
0925 01:23:45.568615 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:45.568620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:45.568704 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:45.568785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:45.570293 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.571020 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.572525 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.572915 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.573021 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:45.576223 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:45.576228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:45.576232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:45.576265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:45.578759 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:45.579411 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:45.579416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:45.579420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:45.579543 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:45.579923 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:45.579928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:45.579931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:45.580198 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:45.580326 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:45.581094 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 01:23:45.581307 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:45.582672 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:45.565626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:45771)
0925 01:23:45.566549 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:45.566603 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:45.568364 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:45.569038 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:45.569046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:45.569537 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:45.570103 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:45.570122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.571160 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.571163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:45.572384 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:45.572397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:45.573134 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:45.573141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:45.573364 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:45.573957 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:45.573984 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:45.576024 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:45.578882 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:45.578891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:45.578909 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:45.579303 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:45.579640 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:45.579644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:45.579648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:45.579821 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:45.580315 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:45.580322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:45.580682 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 01:23:45.581000 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 01:23:45.581527 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":355,"threads_started":1}
I20260925 01:23:45.583904 13699 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:45.584182 13699 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:45.584841 13699 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
I20260925 01:23:45.586223 13243 ts_manager.cc:194] Registered new tserver with Master: bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891)
I20260925 01:23:45.587035 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:45771
I20260925 01:23:45.591006 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.236.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.189.keytab.
Entry for principal oryx/127.6.236.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.189.keytab.
I20260925 01:23:45.634842 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:35469
--webserver_interface=127.6.236.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.189.keytab
--principal=oryx/127.6.236.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:43315,127.6.236.189:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:23:45.650123 13432 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
W20260925 01:23:45.941401 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:45.941694 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:45.941819 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:45.948747 13704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:45.948856 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:45.948932 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:45.949005 13704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:45.949064 13704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:45.958421 13704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:43315,127.6.236.189:35469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.189.keytab
--principal=oryx/127.6.236.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.236.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
W20260925 01:23:45.958767 13704 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.236.190:43315,127.6.236.189:35469'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 01:23:45.959749 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:45.961036 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:45.972134 13709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.972112 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:45.972074 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:45.972775 13704 server_base.cc:1034] running on GCE node
I20260925 01:23:45.973457 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:45.974769 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:45.975984 13704 hybrid_clock.cc:648] HybridClock initialized: now 1790299425975915 us; error 41 us; skew 500 ppm
I20260925 01:23:45.976287 13704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:45 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:45.979280 13704 init.cc:377] Logged in from keytab as oryx/127.6.236.189@KRBTEST.COM (short username oryx)
I20260925 01:23:45.980649 13704 webserver.cc:460] Webserver started at http://127.6.236.189:40753/ using document root <none> and password file <none>
I20260925 01:23:45.981309 13704 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:45.981418 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:45.981710 13704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:45.984536 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/instance:
uuid: "514d558695924e5fb2a9071df8919ebf"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a0c4ab2b5c564bcfa2b2b6d52a78a4e"
server_key_iv: "bdeb8128101958a1c4050764bb1cb28c"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.985248 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal/instance:
uuid: "514d558695924e5fb2a9071df8919ebf"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a0c4ab2b5c564bcfa2b2b6d52a78a4e"
server_key_iv: "bdeb8128101958a1c4050764bb1cb28c"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.990118 13704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:23:45.993415 13719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:45.994719 13704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:45.994853 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "514d558695924e5fb2a9071df8919ebf"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a0c4ab2b5c564bcfa2b2b6d52a78a4e"
server_key_iv: "bdeb8128101958a1c4050764bb1cb28c"
server_key_version: "encryptionkey@0"
I20260925 01:23:45.995023 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.017670 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:46.021811 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:46.022054 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:46.037597 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.189:35469 every 8 connection(s)
I20260925 01:23:46.037626 13704 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.189:35469
I20260925 01:23:46.039131 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
I20260925 01:23:46.041034 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13704
I20260925 01:23:46.043087 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:46.051044 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.189@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:46.062978 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.042988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.6.236.189:35469)
0925 01:23:46.043726 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.043737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.043777 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:23:46.045273 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.045293 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.045679 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.045995 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.047803 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.049097 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.050788 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.051486 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.051630 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.056105 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.056112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.056122 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.056160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.058889 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.059636 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.059642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.059648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.059824 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.060195 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.060199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.060201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.060819 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.061112 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.061544 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.061924 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":342,"threads_started":1}
I20260925 01:23:46.068145 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.053795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.6.236.190:43315)
0925 01:23:46.054017 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.054023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.054615 (+ 592us) server_negotiation.cc:407] Connection header received
0925 01:23:46.054877 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.054884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.054972 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.055081 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.056163 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.056852 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.058260 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.058656 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.058766 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.061983 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.061989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.061994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.062032 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.065430 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.066290 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.066295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.066297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.066438 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.066865 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.066869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.066872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.067193 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.067318 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.067780 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.067932 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:23:46.068441 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.053748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57266)
0925 01:23:46.054470 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.054534 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.054725 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.055126 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.055131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.055769 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.056020 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.056029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.056984 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.056988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.058104 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.058116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.058832 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.058838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.058982 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.059484 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.059504 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.061797 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.065583 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.065594 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.065601 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.066168 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.066576 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.066580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.066583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.066731 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.067322 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.067329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.067499 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.067685 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.067964 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":326,"threads_started":1}
I20260925 01:23:46.069936 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299425, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.189@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
I20260925 01:23:46.080453 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.070527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48100)
0925 01:23:46.070706 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.070724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.070808 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.071078 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.071083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.071457 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.071627 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.071637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.072514 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.072518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.073341 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.073351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.073945 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.073949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.074090 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.074566 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.074581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.076278 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.078644 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.078647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.078650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.078974 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.079288 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.079292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.079294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.079407 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.079817 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.079820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.079926 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.080039 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.080224 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 01:23:46.080641 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.070713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.236.189:35469)
0925 01:23:46.070849 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.070853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.070868 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:23:46.070909 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.070913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.070973 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.071042 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.071753 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.072388 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.073451 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.073831 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.073913 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.076414 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.076418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.076422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.076448 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.078536 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.079068 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.079071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.079073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.079197 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.079486 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.079489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.079491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.079725 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.079826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.080110 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.080345 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":49}
I20260925 01:23:46.084947 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Bootstrap starting.
I20260925 01:23:46.087888 13771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:46.088814 13771 log.cc:824] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:46.091357 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: No bootstrap required, opened a new log
I20260925 01:23:46.100600 13771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } }
I20260925 01:23:46.100914 13771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:46.100986 13771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514d558695924e5fb2a9071df8919ebf, State: Initialized, Role: FOLLOWER
I20260925 01:23:46.101480 13771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } }
I20260925 01:23:46.102190 13777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } } }
I20260925 01:23:46.102526 13777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:23:46.103149 13771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:23:46.110904 13788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:23:46.111013 13788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:23:46.113732 13704 master_runner.cc:428] Detected that this master 514d558695924e5fb2a9071df8919ebf is joining an existing cluster
I20260925 01:23:46.113818 13704 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.236.189:35469
I20260925 01:23:46.123586 13566 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
I20260925 01:23:46.132699 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.117639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.6.236.190:43315)
0925 01:23:46.117823 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.117829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.118710 (+ 881us) server_negotiation.cc:407] Connection header received
0925 01:23:46.118963 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.118969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.119059 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.119124 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.120470 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.121419 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.123314 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.123926 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.124034 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.126754 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.126760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.126765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.126803 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.130164 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.130777 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.130783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.130787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.130913 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.131268 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.131273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.131277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.131560 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.131689 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.132126 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.132339 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:46.132741 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.117694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57270)
0925 01:23:46.118619 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.118638 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.118820 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.119327 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.119331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.119945 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.120311 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.120322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.121613 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.121618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.123142 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.123157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.124376 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.124384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.124561 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.125501 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.125522 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.126570 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.130309 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.130314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.130319 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.130657 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.131021 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.131026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.131029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.131161 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.131670 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.131675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.131850 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.131972 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.132290 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":596,"threads_started":1}
I20260925 01:23:46.146963 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.118727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48112)
0925 01:23:46.133096 (+ 14369us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.133112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.133250 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.133655 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.133662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.134313 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.134528 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.134539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.135893 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.135896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.137099 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.137112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.138332 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.138338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.138487 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.139016 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.139038 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.140042 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.144251 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.144256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.144260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.144769 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.145133 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.145138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.145142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.145291 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.145947 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.145952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.146087 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.146252 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.146552 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14261,"threads_started":1}
I20260925 01:23:46.148069 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.118486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.6.236.189:35469)
0925 01:23:46.118913 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.118919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.133168 (+ 14249us) server_negotiation.cc:407] Connection header received
0925 01:23:46.133359 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.133363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.133430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.133490 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.134668 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.135714 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.137232 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.137897 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.137986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.140244 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.140250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.140255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.140287 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.144102 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.144879 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.144885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.144888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.145014 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.145385 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.145390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.145394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.145770 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.147319 (+ 1549us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.147481 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.147718 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for oryx/127.6.236.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.188@KRBTEST.COM" created.
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.190@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
I20260925 01:23:46.163659 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.139738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.6.236.189:35469)
0925 01:23:46.148715 (+ 8977us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.148721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.149540 (+ 819us) server_negotiation.cc:407] Connection header received
0925 01:23:46.149864 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.149873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.149942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.150027 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.151373 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.152189 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.154332 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.154749 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.154857 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.157946 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.157951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.157954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.157982 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.160614 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.161355 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.161361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.161365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.161548 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.161946 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.161951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.161954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.162368 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.162532 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.163014 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.163250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8875,"thread_start_us":2693,"threads_started":1}
I20260925 01:23:46.163671 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.139428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48126)
0925 01:23:46.149404 (+ 9976us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.149445 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.149670 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.150087 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.150095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.150969 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.151240 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.151250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.152311 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.152317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.153606 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.153621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.154906 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.154913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.155102 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.155701 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.155719 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.157763 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.160738 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.160752 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.160760 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.161230 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.161663 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.161668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.161672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.161834 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.162505 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.162514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.162713 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.162919 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.163268 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9764,"thread_start_us":9625,"threads_started":1}
I20260925 01:23:46.165956 13243 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.236.189:35469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "514d558695924e5fb2a9071df8919ebf"
member_type: NON_VOTER
last_known_addr {
host: "127.6.236.189"
port: 35469
}
attrs {
promote: true
}
}
dest_uuid: "38b03a55932f4ed2a155f9c851b60407"
cas_config_opid_index: -1
I20260925 01:23:46.167192 13243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } }
I20260925 01:23:46.177002 13801 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 514d558695924e5fb2a9071df8919ebf (127.6.236.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } }
I20260925 01:23:46.178610 13746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:46.181656 13801 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.236.189:35469
I20260925 01:23:46.182264 13802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } }
I20260925 01:23:46.182610 13704 master.cc:537] Master@127.6.236.189:35469 shutting down...
I20260925 01:23:46.182552 13746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 01:23:46.182643 13802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.188.keytab.
Entry for principal oryx/127.6.236.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.188.keytab.
I20260925 01:23:46.185132 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:36491
--webserver_interface=127.6.236.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.188.keytab
--principal=oryx/127.6.236.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:23:46.186055 13802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:23:46.188987 13228 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 01:23:46.194262 13704 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 01:23:46.194394 13228 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 -> Peer 514d558695924e5fb2a9071df8919ebf (127.6.236.189:35469): Couldn't send request to peer 514d558695924e5fb2a9071df8919ebf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:23:46.194686 13704 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:46.194869 13704 tablet_replica.cc:330] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: stopping tablet replica
I20260925 01:23:46.201437 13704 master.cc:552] Master@127.6.236.189:35469 shutdown complete.
I20260925 01:23:46.201546 13704 master_runner.cc:305] Clearing existing system tablet
I20260925 01:23:46.201982 13704 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:46.202075 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:46.205091 13704 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 01:23:46.206955 13807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:46.207346 13704 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.208086 13704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 01:23:46.208182 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "514d558695924e5fb2a9071df8919ebf"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a0c4ab2b5c564bcfa2b2b6d52a78a4e"
server_key_iv: "bdeb8128101958a1c4050764bb1cb28c"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.213032 13704 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:46.219702 13704 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:23:46.219867 13704 log.cc:1196] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 01:23:46.220264 13704 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Deleting consensus metadata
I20260925 01:23:46.220506 13704 master_runner.cc:315] Copying system tablet from 127.6.236.190:43315
I20260925 01:23:46.225586 13704 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: tablet copy: Beginning tablet copy session from remote peer at address 127.6.236.190:43315
I20260925 01:23:46.244701 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.226354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.6.236.190:43315)
0925 01:23:46.226533 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.226539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.230752 (+ 4213us) server_negotiation.cc:407] Connection header received
0925 01:23:46.230932 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.230938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.231041 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.231153 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.232139 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.233237 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.235115 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.235761 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.235895 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.238017 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.238024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.238029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.238069 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.241630 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.242414 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.242420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.242424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.242602 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.243124 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.243130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.243133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.243535 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.243711 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.244136 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.244367 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:46.244737 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.226324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57280)
0925 01:23:46.230624 (+ 4300us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.230643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.230806 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.231191 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.231196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.231717 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.231961 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.231982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.233400 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.233406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.234949 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.234965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.235923 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.235929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.236131 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.236793 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.236817 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.237803 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.241772 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.241777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.241782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.242297 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.242716 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.242722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.242725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.242899 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.243666 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.243672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.243843 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.243999 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.244303 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4162,"thread_start_us":4071,"threads_started":1}
I20260925 01:23:46.245867 13263 tablet_copy_service.cc:140] P 38b03a55932f4ed2a155f9c851b60407: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 514d558695924e5fb2a9071df8919ebf ({username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:57280)
I20260925 01:23:46.246165 13263 tablet_copy_service.cc:161] P 38b03a55932f4ed2a155f9c851b60407: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 514d558695924e5fb2a9071df8919ebf at {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:57280: session id = 514d558695924e5fb2a9071df8919ebf-00000000000000000000000000000000
I20260925 01:23:46.250408 13263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:23:46.252542 13704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:23:46.259097 13704 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: tablet copy: Starting download of 0 data blocks...
I20260925 01:23:46.259387 13704 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: tablet copy: Starting download of 1 WAL segments...
I20260925 01:23:46.261994 13704 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:23:46.266325 13263 tablet_copy_service.cc:342] P 38b03a55932f4ed2a155f9c851b60407: Request end of tablet copy session 514d558695924e5fb2a9071df8919ebf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:57280
I20260925 01:23:46.266525 13263 tablet_copy_service.cc:433] P 38b03a55932f4ed2a155f9c851b60407: ending tablet copy session 514d558695924e5fb2a9071df8919ebf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 514d558695924e5fb2a9071df8919ebf
I20260925 01:23:46.276913 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:46.279474 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:46.279874 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:46.291868 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:46.295168 13704 server_base.cc:1034] running on GCE node
I20260925 01:23:46.295581 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:46.296232 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:46.297514 13704 hybrid_clock.cc:648] HybridClock initialized: now 1790299426297448 us; error 30 us; skew 500 ppm
I20260925 01:23:46.297739 13704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:46.299823 13704 init.cc:377] Logged in from keytab as oryx/127.6.236.189@KRBTEST.COM (short username oryx)
I20260925 01:23:46.301548 13704 webserver.cc:460] Webserver started at http://127.6.236.189:34443/ using document root <none> and password file <none>
I20260925 01:23:46.301925 13704 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:46.302006 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:46.305083 13704 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 01:23:46.306954 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:46.307376 13704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 01:23:46.307540 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "514d558695924e5fb2a9071df8919ebf"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a0c4ab2b5c564bcfa2b2b6d52a78a4e"
server_key_iv: "bdeb8128101958a1c4050764bb1cb28c"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.307703 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.345804 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:46.349721 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:46.371318 13704 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.189:35469
I20260925 01:23:46.371347 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.189:35469 every 8 connection(s)
I20260925 01:23:46.372406 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
I20260925 01:23:46.377686 13876 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:23:46.381529 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Bootstrap starting.
I20260925 01:23:46.398509 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:23:46.399009 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Bootstrap complete.
I20260925 01:23:46.400094 13876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } }
I20260925 01:23:46.400367 13876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 514d558695924e5fb2a9071df8919ebf, State: Initialized, Role: LEARNER
I20260925 01:23:46.400665 13876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } }
I20260925 01:23:46.401494 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } }
I20260925 01:23:46.401790 13876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:46.401932 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: LEARNER
I20260925 01:23:46.409636 13890 catalog_manager.cc:1259] Loaded cluster ID: 6a364ac311484adf97fb375bc795e735
I20260925 01:23:46.409804 13890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: loading cluster ID for follower catalog manager: success
I20260925 01:23:46.413466 13890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: acquiring CA information for follower catalog manager: success
I20260925 01:23:46.416028 13890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:23:46.553086 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:46.553359 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:46.553486 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:46.560901 13805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:46.561035 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:46.561111 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:46.561192 13805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:46.561257 13805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:46.573323 13805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.188.keytab
--principal=oryx/127.6.236.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.236.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:46.574635 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:46.575862 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:46.586697 13896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:46.587030 13895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:46.586776 13898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:46.587725 13805 server_base.cc:1034] running on GCE node
I20260925 01:23:46.588433 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:46.589299 13699 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
I20260925 01:23:46.589852 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:46.591075 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790299426591014 us; error 53 us; skew 500 ppm
I20260925 01:23:46.591398 13805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:46.594277 13805 init.cc:377] Logged in from keytab as oryx/127.6.236.188@KRBTEST.COM (short username oryx)
I20260925 01:23:46.595621 13805 webserver.cc:460] Webserver started at http://127.6.236.188:44947/ using document root <none> and password file <none>
I20260925 01:23:46.596253 13805 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:46.596369 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:46.596654 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:46.599473 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/instance:
uuid: "b4f8299c682c4c3891d02156d15c5937"
format_stamp: "Formatted at 2026-09-25 01:23:46 on dist-test-slave-s0b2"
server_key: "b8581286ca3bcda79d21e1243d111ff5"
server_key_iv: "45fb896fd0979b8e805c30f43c88d3df"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.600235 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal/instance:
uuid: "b4f8299c682c4c3891d02156d15c5937"
format_stamp: "Formatted at 2026-09-25 01:23:46 on dist-test-slave-s0b2"
server_key: "b8581286ca3bcda79d21e1243d111ff5"
server_key_iv: "45fb896fd0979b8e805c30f43c88d3df"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.605510 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:23:46.608940 13905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:46.610375 13805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:46.610543 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "b4f8299c682c4c3891d02156d15c5937"
format_stamp: "Formatted at 2026-09-25 01:23:46 on dist-test-slave-s0b2"
server_key: "b8581286ca3bcda79d21e1243d111ff5"
server_key_iv: "45fb896fd0979b8e805c30f43c88d3df"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.610723 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.632295 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:46.635951 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:46.636188 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:46.651732 13805 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.188:36491
I20260925 01:23:46.651751 13956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.188:36491 every 8 connection(s)
I20260925 01:23:46.653067 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
I20260925 01:23:46.656679 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:46.659586 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 13805
I20260925 01:23:46.663733 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.188@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:46.680344 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.667238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.6.236.190:43315)
0925 01:23:46.667388 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.667392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.667950 (+ 558us) server_negotiation.cc:407] Connection header received
0925 01:23:46.668115 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.668121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.668188 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.668267 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.669366 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.670098 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.671588 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.672021 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.672128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.675326 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.675331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.675334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.675366 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.677725 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.678408 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.678414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.678418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.678542 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.678898 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.678901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.678903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.679179 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.679315 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.679816 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.680050 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:23:46.681246 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.667193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57296)
0925 01:23:46.667821 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.667856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.668000 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.668343 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.668349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.668953 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.669225 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.669235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.670233 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.670259 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.671453 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.671474 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.672229 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.672237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.672382 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.672910 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.672935 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.675144 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.677864 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.677875 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.677886 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.678305 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.678654 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.678659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.678662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.678779 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.679299 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.679306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.679489 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.679689 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.680120 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
I20260925 01:23:46.681264 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.661641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.6.236.188:36491)
0925 01:23:46.662376 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.662390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.662436 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:23:46.664449 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.664477 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.664899 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.665273 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.667014 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.668713 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.669897 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.670578 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.670746 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.674774 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.674783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.674793 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.674825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.677426 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.678000 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.678006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.678008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.678161 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.678520 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.678523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.678525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.678986 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.679255 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.679657 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.680214 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":495,"thread_start_us":292,"threads_started":1}
I20260925 01:23:46.682797 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.188@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
I20260925 01:23:46.697113 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.683577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48136)
0925 01:23:46.683769 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.683788 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.683888 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.686909 (+ 3021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.686912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.687209 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.687332 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.687339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.688576 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.688579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.689482 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.689492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.690091 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.690095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.690220 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.690739 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.690756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.692499 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.695196 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.695199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.695202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.695561 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.695871 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.695875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.695877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.695998 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.696496 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.696499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.696605 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.696701 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.696887 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 01:23:46.697180 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.683532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.6.236.189:35469)
0925 01:23:46.686647 (+ 3115us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.686651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.686667 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:46.686715 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.686718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.686774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.686856 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.687461 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.688446 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.689607 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.689977 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.690064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.692659 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.692663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.692666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.692696 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.695081 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.695642 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.695645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.695647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.695779 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.696123 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.696128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.696130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.696403 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.696544 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.696773 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.696928 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3007,"thread_start_us":2896,"threads_started":1}
I20260925 01:23:46.698261 13957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.188@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
I20260925 01:23:46.709304 13961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.698851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.0.0.1:33544)
0925 01:23:46.699040 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.699054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.699140 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.699375 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.699378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.699704 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.699834 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.699841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.700716 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.700721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.701521 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.701532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.702191 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.702196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.702392 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.702924 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.702940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.704819 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.707278 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.707282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.707284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.707644 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.707955 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.707959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.707961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.708125 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.708592 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.708595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.708726 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.708828 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.708998 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 01:23:46.709362 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.699031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.6.236.188:36491)
0925 01:23:46.699167 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.699172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.699187 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:23:46.699225 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.699229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.699279 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.699347 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.699951 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.700609 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.701638 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.702060 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.702166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.705000 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.705004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.705007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.705034 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.707155 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.707733 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.707736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.707738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.707848 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.708198 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.708201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.708203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.708497 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.708598 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.708903 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.709114 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":45}
I20260925 01:23:46.713824 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Bootstrap starting.
I20260925 01:23:46.717060 13957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:46.718062 13957 log.cc:824] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:46.720481 13957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: No bootstrap required, opened a new log
I20260925 01:23:46.727326 13957 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } }
I20260925 01:23:46.727610 13957 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:46.727716 13957 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f8299c682c4c3891d02156d15c5937, State: Initialized, Role: FOLLOWER
I20260925 01:23:46.728235 13957 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } }
I20260925 01:23:46.729012 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } } }
I20260925 01:23:46.729287 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:23:46.729899 13957 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:46.735924 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13304
W20260925 01:23:46.738523 13975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:23:46.738706 13975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:23:46.742873 13805 master_runner.cc:428] Detected that this master b4f8299c682c4c3891d02156d15c5937 is joining an existing cluster
I20260925 01:23:46.742985 13805 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.236.188:36491
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.129@KRBTEST.COM
I20260925 01:23:46.758217 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.737043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.6.236.129:42755)
0925 01:23:46.738074 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.738083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.738110 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:23:46.738189 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.738197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.738457 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.740277 (+ 1820us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.740496 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.741949 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.743531 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.744393 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.745143 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.750225 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.750268 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.750287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.750336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.754080 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.754943 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.754952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.754956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.755122 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.755605 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.755609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.755613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.756055 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.757384 (+ 1329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.757547 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.757804 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":743,"threads_started":1}
I20260925 01:23:46.759917 13427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:46.774096 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.753346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.6.236.190:43315)
0925 01:23:46.753543 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.753549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.759123 (+ 5574us) server_negotiation.cc:407] Connection header received
0925 01:23:46.759388 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.759395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.759477 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.759546 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.760994 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.762116 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.764162 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.764790 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.764887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.767689 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.767697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.767702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.767741 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.771072 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.771789 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.771795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.771798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.771944 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.772456 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.772461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.772465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.772817 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.772948 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.773534 (+ 586us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.773759 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:46.775558 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.752567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57312)
0925 01:23:46.759014 (+ 6447us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.759042 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.759238 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.759749 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.759754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.760455 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.760818 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.760832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.762306 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.762310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.763983 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.764001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.765179 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.765186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.765397 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.766388 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.766413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.767471 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.771220 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.771225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.771230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.771651 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.772098 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.772103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.772106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.772330 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.773036 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.773042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.773254 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.774326 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.775108 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6301,"mutex_wait_us":3,"thread_start_us":6966,"threads_started":1}
I20260925 01:23:46.777667 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.753142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.6.236.189:35469)
0925 01:23:46.753325 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.753331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.760445 (+ 7114us) server_negotiation.cc:407] Connection header received
0925 01:23:46.760711 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.760717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.760818 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.760886 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.762319 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.763430 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.766102 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.766851 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.766972 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.770018 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.770026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.770031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.770074 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.773573 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.775237 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.775243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.775247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.775412 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.775931 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.775937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.775940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.776298 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.776427 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.777057 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.777310 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:23:46.778995 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.752749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48150)
0925 01:23:46.760335 (+ 7586us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.760353 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.760551 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.761092 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.761097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.761810 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.762112 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.762124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.763647 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.763653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.765950 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.765964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.767291 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.767298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.767474 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.768132 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.768155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.769765 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.773723 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.773728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.773747 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.775070 (+ 1323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.775588 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.775594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.775598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.775793 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.776524 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.776529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.776759 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.778139 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.778501 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7428,"mutex_wait_us":546,"thread_start_us":12552,"threads_started":1}
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.190@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
I20260925 01:23:46.800007 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.765820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.0.0.1:33556)
0925 01:23:46.772654 (+ 6834us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.772672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.773064 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.773660 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.773665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.785678 (+ 12013us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.786582 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.786594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.788137 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.788142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.789792 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.789807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.791003 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.791009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.791194 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.791838 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.792213 (+ 375us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.793354 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.797229 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.797233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.797237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.797754 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.798145 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.798149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.798153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.798319 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.798925 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.798930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.799081 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.799209 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.799512 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6699,"spinlock_wait_cycles":26368,"thread_start_us":8331,"threads_started":1}
I20260925 01:23:46.801039 13960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.765820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.6.236.188:36491)
0925 01:23:46.765966 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.765973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.772747 (+ 6774us) server_negotiation.cc:407] Connection header received
0925 01:23:46.773331 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.773337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.773408 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.773471 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.786729 (+ 13258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.787918 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.789926 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.790622 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.790713 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.793515 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.793520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.793524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.793553 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.797111 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.797865 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.797870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.797873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.798036 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.798418 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.798423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.798426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.798734 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.798881 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.800409 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.800667 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":896}
I20260925 01:23:46.805248 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.781923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.6.236.188:36491)
0925 01:23:46.786371 (+ 4448us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.786377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.786568 (+ 191us) server_negotiation.cc:407] Connection header received
0925 01:23:46.786851 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.786857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.786958 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.787037 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.788327 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.789632 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.791341 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.792020 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.792124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.796345 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.796352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.796357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.796400 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.800400 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.803032 (+ 2632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.803038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.803041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.803193 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.803619 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.803624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.803628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.803974 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.804098 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.804664 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.804860 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4295,"thread_start_us":4168,"threads_started":1}
I20260925 01:23:46.806716 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.781043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.0.0.1:33564)
0925 01:23:46.786477 (+ 5434us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.786497 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.786660 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.787436 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.787441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.787874 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.788111 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.788124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.789830 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.789836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.791178 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.791193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.792362 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.792368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.792535 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.793125 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.793150 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.796098 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.800561 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.800566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.800570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.802886 (+ 2316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.803335 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.803340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.803343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.803492 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.804180 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.804186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.804386 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.805781 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.806081 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4934,"thread_start_us":4792,"threads_started":1}
I20260925 01:23:46.808316 13243 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.236.188:36491: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b4f8299c682c4c3891d02156d15c5937"
member_type: NON_VOTER
last_known_addr {
host: "127.6.236.188"
port: 36491
}
attrs {
promote: true
}
}
dest_uuid: "38b03a55932f4ed2a155f9c851b60407"
cas_config_opid_index: 5
I20260925 01:23:46.809051 13243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } }
I20260925 01:23:46.814857 13988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b4f8299c682c4c3891d02156d15c5937 (127.6.236.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.815719 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:46.818367 13988 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.236.188:36491
I20260925 01:23:46.820629 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.820940 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:46.827965 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.819960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48156)
0925 01:23:46.821312 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.821331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.821483 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.821867 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.821872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.821900 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:46.822093 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.822104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.823471 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.823477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.825533 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.825550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.827203 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.827209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.827405 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.827464 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.827560 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.827704 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235}
I20260925 01:23:46.827971 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.820248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.6.236.189:35469)
0925 01:23:46.820442 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.820449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.821401 (+ 952us) server_negotiation.cc:407] Connection header received
0925 01:23:46.821615 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.821622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.821711 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.821820 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:46.822276 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.823338 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.825677 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.827013 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.827146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.827361 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.827582 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.827725 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:23:46.828958 13851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 LEARNER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:23:46.829589 13989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 01:23:46.832391 13851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b4f8299c682c4c3891d02156d15c5937 (127.6.236.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.838694 13993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.839165 13993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: LEARNER
I20260925 01:23:46.839430 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b03a55932f4ed2a155f9c851b60407. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: NON_VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: true } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.839704 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: LEARNER
I20260925 01:23:46.840267 13989 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: attempting to promote NON_VOTER 514d558695924e5fb2a9071df8919ebf to VOTER
I20260925 01:23:46.841004 13989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } }
I20260925 01:23:46.842880 13851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 LEARNER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 01:23:46.843374 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 01:23:46.845746 13988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 514d558695924e5fb2a9071df8919ebf (127.6.236.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.846544 13805 master.cc:537] Master@127.6.236.188:36491 shutting down...
I20260925 01:23:46.847285 13850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 514d558695924e5fb2a9071df8919ebf (127.6.236.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.848374 13932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:23:46.849475 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.849743 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:46.850299 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.850534 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:46.850661 13931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 01:23:46.851801 13988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:46.858825 13879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:46.859261 13879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: FOLLOWER
W20260925 01:23:46.862180 13228 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 -> Peer b4f8299c682c4c3891d02156d15c5937 (127.6.236.188:36491): Couldn't send request to peer b4f8299c682c4c3891d02156d15c5937. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:23:46.864598 13805 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:46.865087 13805 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:46.865386 13805 tablet_replica.cc:330] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: stopping tablet replica
I20260925 01:23:46.873168 13805 master.cc:552] Master@127.6.236.188:36491 shutdown complete.
I20260925 01:23:46.873301 13805 master_runner.cc:305] Clearing existing system tablet
I20260925 01:23:46.873734 13805 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:46.873888 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:46.876873 13805 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:46.878554 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:46.878882 13805 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.879293 13805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 01:23:46.879441 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "b4f8299c682c4c3891d02156d15c5937"
format_stamp: "Formatted at 2026-09-25 01:23:46 on dist-test-slave-s0b2"
server_key: "b8581286ca3bcda79d21e1243d111ff5"
server_key_iv: "45fb896fd0979b8e805c30f43c88d3df"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.884379 13805 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:46.891088 13805 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:23:46.891371 13805 log.cc:1196] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 01:23:46.891878 13805 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Deleting consensus metadata
I20260925 01:23:46.892215 13805 master_runner.cc:315] Copying system tablet from 127.6.236.190:43315
I20260925 01:23:46.894744 13805 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: tablet copy: Beginning tablet copy session from remote peer at address 127.6.236.190:43315
I20260925 01:23:46.912688 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.895639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.6.236.190:43315)
0925 01:23:46.895842 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:46.895848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:46.896286 (+ 438us) server_negotiation.cc:407] Connection header received
0925 01:23:46.896642 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:46.896649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:46.896748 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:46.896821 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:46.898035 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.899247 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.901222 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.901868 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.901961 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.905101 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:46.905109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:46.905115 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:46.905184 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:46.909444 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.910290 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.910296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.910300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.910461 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:46.910979 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:46.910984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:46.910988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:46.911395 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:46.911528 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:46.912096 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 01:23:46.912326 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:23:46.913962 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:46.895544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57316)
0925 01:23:46.896174 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:46.896196 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:46.896399 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:46.897037 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:46.897042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:46.897562 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:46.897841 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.897853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.899478 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.899483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.901062 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:46.901078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:46.902289 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:46.902295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:46.902468 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:46.903207 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:46.903232 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:46.904781 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:46.909596 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.909601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.909605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.910087 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.910608 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:46.910613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:46.910616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:46.910848 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:46.911620 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:46.911625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:46.911798 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 01:23:46.913204 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0925 01:23:46.913504 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":410,"threads_started":1}
I20260925 01:23:46.914681 13263 tablet_copy_service.cc:140] P 38b03a55932f4ed2a155f9c851b60407: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b4f8299c682c4c3891d02156d15c5937 ({username='oryx', principal='oryx/127.6.236.188@KRBTEST.COM'} at 127.0.0.1:57316)
I20260925 01:23:46.914873 13263 tablet_copy_service.cc:161] P 38b03a55932f4ed2a155f9c851b60407: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b4f8299c682c4c3891d02156d15c5937 at {username='oryx', principal='oryx/127.6.236.188@KRBTEST.COM'} at 127.0.0.1:57316: session id = b4f8299c682c4c3891d02156d15c5937-00000000000000000000000000000000
I20260925 01:23:46.914915 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13304
I20260925 01:23:46.918170 13263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:23:46.924263 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:23:46.931052 13805 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: tablet copy: Starting download of 0 data blocks...
I20260925 01:23:46.931324 13805 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: tablet copy: Starting download of 1 WAL segments...
I20260925 01:23:46.933537 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:42755
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:43315
--webserver_port=40411
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:23:46.933638 13805 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:23:46.936856 13263 tablet_copy_service.cc:342] P 38b03a55932f4ed2a155f9c851b60407: Request end of tablet copy session b4f8299c682c4c3891d02156d15c5937-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.236.188@KRBTEST.COM'} at 127.0.0.1:57316
I20260925 01:23:46.936991 13263 tablet_copy_service.cc:433] P 38b03a55932f4ed2a155f9c851b60407: ending tablet copy session b4f8299c682c4c3891d02156d15c5937-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b4f8299c682c4c3891d02156d15c5937
I20260925 01:23:46.947173 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:46.949582 14004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:46.949927 14005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:46.956892 13805 server_base.cc:1034] running on GCE node
W20260925 01:23:46.957469 14007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:46.957916 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:46.958540 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:46.959764 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790299426959713 us; error 40 us; skew 500 ppm
I20260925 01:23:46.959973 13805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:46 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299426, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:46.961973 13805 init.cc:377] Logged in from keytab as oryx/127.6.236.188@KRBTEST.COM (short username oryx)
I20260925 01:23:46.967062 13805 webserver.cc:460] Webserver started at http://127.6.236.188:40837/ using document root <none> and password file <none>
I20260925 01:23:46.967338 13805 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:46.967425 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:46.969882 13805 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 01:23:46.971513 14014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:46.971899 13805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 01:23:46.972019 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "b4f8299c682c4c3891d02156d15c5937"
format_stamp: "Formatted at 2026-09-25 01:23:46 on dist-test-slave-s0b2"
server_key: "b8581286ca3bcda79d21e1243d111ff5"
server_key_iv: "45fb896fd0979b8e805c30f43c88d3df"
server_key_version: "encryptionkey@0"
I20260925 01:23:46.972177 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:46.984715 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:46.987818 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:47.023983 13805 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.188:36491
I20260925 01:23:47.024000 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.188:36491 every 8 connection(s)
I20260925 01:23:47.024924 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
I20260925 01:23:47.030510 14066 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:23:47.034780 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Bootstrap starting.
I20260925 01:23:47.050586 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:23:47.051020 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: Bootstrap complete.
I20260925 01:23:47.051831 14066 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } }
I20260925 01:23:47.052140 14066 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4f8299c682c4c3891d02156d15c5937, State: Initialized, Role: LEARNER
I20260925 01:23:47.052405 14066 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } }
I20260925 01:23:47.054675 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:47.054997 14066 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:47.055157 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: LEARNER
I20260925 01:23:47.065058 14080 catalog_manager.cc:1259] Loaded cluster ID: 6a364ac311484adf97fb375bc795e735
I20260925 01:23:47.065132 14080 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: loading cluster ID for follower catalog manager: success
I20260925 01:23:47.067520 14080 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: acquiring CA information for follower catalog manager: success
I20260925 01:23:47.069101 14080 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:23:47.289615 14003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:47.289856 14003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:47.289944 14003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:47.296931 14003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:47.297027 14003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:47.297144 14003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:47.306583 14003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:42755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=40411
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:47.307874 14003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:47.309185 14003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:23:47.322682 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.314900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.0.0.1:33574)
0925 01:23:47.315102 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.315147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.315305 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.316206 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.316213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.316251 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:47.316465 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.316477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.318320 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.318327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.320293 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.320311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.321848 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.321856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.322071 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.322112 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.322213 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.322380 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 01:23:47.322721 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.315101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.6.236.188:36491)
0925 01:23:47.315821 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.315826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.315851 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:23:47.315927 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.315932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.316020 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.316150 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:47.316656 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.318131 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.320455 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.321680 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.321823 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.322034 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.322272 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.322408 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":441,"threads_started":1}
W20260925 01:23:47.322638 14086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:47.323854 14041 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
W20260925 01:23:47.324419 14089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:47.324484 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b03a55932f4ed2a155f9c851b60407. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:47.324777 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: LEARNER
W20260925 01:23:47.324959 14087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:47.325838 14003 server_base.cc:1034] running on GCE node
I20260925 01:23:47.326460 14003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:47.327376 14003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:47.328569 14003 hybrid_clock.cc:648] HybridClock initialized: now 1790299427328518 us; error 36 us; skew 500 ppm
I20260925 01:23:47.328859 14003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:47.331722 14003 init.cc:377] Logged in from keytab as oryx/127.6.236.129@KRBTEST.COM (short username oryx)
I20260925 01:23:47.333084 14003 webserver.cc:460] Webserver started at http://127.6.236.129:40411/ using document root <none> and password file <none>
I20260925 01:23:47.333750 14003 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:47.333875 14003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:47.339040 14003 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:47.341635 14096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.342935 14003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:23:47.343099 14003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
format_stamp: "Formatted at 2026-09-25 01:23:44 on dist-test-slave-s0b2"
server_key: "a434b62f519e288e00abb7f4718a7571"
server_key_iv: "915d6d80e8965d262dce022a9399f3ed"
server_key_version: "encryptionkey@0"
I20260925 01:23:47.344029 14003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:47.356395 14003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:47.360019 14003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:47.360252 14003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:47.361114 14003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:47.362378 14003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:47.362471 14003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.362571 14003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:47.362640 14003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.396663 14003 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42755
I20260925 01:23:47.396727 14208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42755 every 8 connection(s)
I20260925 01:23:47.398066 14003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:47.398581 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14003
I20260925 01:23:47.398897 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13438
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.130@KRBTEST.COM
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
I20260925 01:23:47.430081 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.401036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:49811 (local address 127.6.236.188:36491)
0925 01:23:47.401245 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.401252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.403161 (+ 1909us) server_negotiation.cc:407] Connection header received
0925 01:23:47.408711 (+ 5550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.408717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.408801 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.408874 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.412680 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.413752 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.416953 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.417616 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.417711 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.422182 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.422189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.422193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.422279 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.425648 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427297 (+ 1649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.427302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.427306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.427466 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428008 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.428014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.428017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.428370 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.428502 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.429518 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.429760 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:23:47.430132 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.404001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:43447 (local address 127.6.236.189:35469)
0925 01:23:47.407804 (+ 3803us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.407811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.410202 (+ 2391us) server_negotiation.cc:407] Connection header received
0925 01:23:47.410372 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.410377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.410464 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.410565 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.412673 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.413715 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.416655 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.417301 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.417399 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.422975 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.422983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.422987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.423021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.426876 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427735 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.427741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.427744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.427903 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428342 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.428346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.428349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.428695 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.428827 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.429508 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.429759 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3664,"thread_start_us":3521,"threads_started":1}
I20260925 01:23:47.430271 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.401778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55233 (local address 127.6.236.190:43315)
0925 01:23:47.401926 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.401934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.408408 (+ 6474us) server_negotiation.cc:407] Connection header received
0925 01:23:47.408660 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.408666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.408743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.408819 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.412789 (+ 3970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.413881 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.416654 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.417279 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.417390 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.422541 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.422549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.422554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.422598 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.425786 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428155 (+ 2369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.428160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.428164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.428301 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428724 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.428728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.428731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.429097 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.429239 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.429712 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.429927 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 01:23:47.430431 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.400316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.6.236.130:43397)
0925 01:23:47.407323 (+ 7007us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.407331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.407357 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:23:47.407431 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.407439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.407672 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.409716 (+ 2044us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.409938 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.411391 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.413231 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.414093 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.420413 (+ 6320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.422221 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.422262 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.422282 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.422323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.426352 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427191 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.427201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.427205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.427354 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427810 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.427815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.427819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.428265 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.428533 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.429783 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.430044 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6828,"thread_start_us":6760,"threads_started":1}
I20260925 01:23:47.431869 13561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:47.432348 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.401498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.6.236.129:49811)
0925 01:23:47.403007 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.403051 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.408358 (+ 5307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.409361 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.409372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.411488 (+ 2116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.412461 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.412489 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.413932 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.413937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.416805 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.416817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.418002 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.418009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.418201 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.419015 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.419054 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.421923 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.425862 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.425871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.425890 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.427175 (+ 1285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.427591 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.427599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.427853 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.428593 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.428600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.429070 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.430900 (+ 1830us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.431170 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":797,"spinlock_wait_cycles":19200,"thread_start_us":930,"threads_started":1}
I20260925 01:23:47.432387 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.401491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:55233)
0925 01:23:47.408292 (+ 6801us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.408316 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.408504 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.409360 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.409372 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.411498 (+ 2126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.412632 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.412644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.414047 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.414052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.416476 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.416489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.417772 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.417779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.418210 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.419019 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.419055 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.422293 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.426057 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.426062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.426066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.428027 (+ 1961us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.428442 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.428449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.428604 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.429313 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.429317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.429452 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.430569 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.431061 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6637,"mutex_wait_us":990,"spinlock_wait_cycles":40320,"thread_start_us":7838,"threads_started":1}
I20260925 01:23:47.432493 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.403629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.6.236.129:43447)
0925 01:23:47.410108 (+ 6479us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.410125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.410227 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.410589 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.410594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.411503 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.412462 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.412487 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.413928 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.413932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.416471 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.416485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.417771 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.417778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.418187 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.419182 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.419205 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.422709 (+ 3504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.427054 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.427057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.427059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.427610 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.428039 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.428044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.428047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.428209 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.428907 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.428912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.429069 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.430349 (+ 1280us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.430904 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6312,"mutex_wait_us":229,"spinlock_wait_cycles":41088,"thread_start_us":4380,"threads_started":1}
I20260925 01:23:47.434370 14209 heartbeater.cc:347] Connected to a master server at 127.6.236.188:36491
I20260925 01:23:47.434701 14211 heartbeater.cc:347] Connected to a master server at 127.6.236.189:35469
I20260925 01:23:47.434748 14209 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:47.434832 14211 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:47.434958 14210 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:47.435081 14210 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:47.435784 14209 heartbeater.cc:511] Master 127.6.236.188:36491 requested a full tablet report, sending...
I20260925 01:23:47.437037 14211 heartbeater.cc:511] Master 127.6.236.189:35469 requested a full tablet report, sending...
I20260925 01:23:47.437108 14210 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
I20260925 01:23:47.438187 14031 ts_manager.cc:194] Registered new tserver with Master: e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:47.438781 13841 ts_manager.cc:194] Registered new tserver with Master: e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:47.439721 13243 ts_manager.cc:194] Re-registered known tserver with Master: e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:47.440527 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:55233
I20260925 01:23:47.515954 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13438
I20260925 01:23:47.529453 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:43397
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:43315
--webserver_port=40317
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:23:47.729848 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: NON_VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: true } } }
I20260925 01:23:47.730182 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:47.731868 14221 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: attempting to promote NON_VOTER b4f8299c682c4c3891d02156d15c5937 to VOTER
I20260925 01:23:47.732406 14221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } }
I20260925 01:23:47.734197 13850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:23:47.734311 14041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 LEARNER]: Refusing update from remote peer 38b03a55932f4ed2a155f9c851b60407: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:23:47.734851 14222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:23:47.735399 13988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:23:47.738438 13988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8299c682c4c3891d02156d15c5937 (127.6.236.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.739147 13850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8299c682c4c3891d02156d15c5937 (127.6.236.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.741997 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.742295 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:47.742447 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.742722 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:47.743136 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.743342 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:47.742925 14041 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b4f8299c682c4c3891d02156d15c5937 (127.6.236.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.743992 14225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.744295 14225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:23:47.746049 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38b03a55932f4ed2a155f9c851b60407" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:47.746295 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 01:23:47.844254 14220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:47.844542 14220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:47.844676 14220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:47.851881 14220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:47.851999 14220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:47.852145 14220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:47.862120 14220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:43397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=40317
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:47.863402 14220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:47.864694 14220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:47.877255 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:47.877331 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:47.877254 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:47.878419 14220 server_base.cc:1034] running on GCE node
I20260925 01:23:47.878993 14220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:47.879866 14220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:47.881070 14220 hybrid_clock.cc:648] HybridClock initialized: now 1790299427881018 us; error 37 us; skew 500 ppm
I20260925 01:23:47.881353 14220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:47.884308 14220 init.cc:377] Logged in from keytab as oryx/127.6.236.130@KRBTEST.COM (short username oryx)
I20260925 01:23:47.885653 14220 webserver.cc:460] Webserver started at http://127.6.236.130:40317/ using document root <none> and password file <none>
I20260925 01:23:47.886332 14220 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:47.886457 14220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:47.891500 14220 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:23:47.894016 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.895318 14220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:23:47.895457 14220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "08e99fa0e0a040878a25b6de75fc6871"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "fbc6a49279bdfb5b91464de79d602689"
server_key_iv: "f3f70c35e3e3f2ee60355c210786ba5f"
server_key_version: "encryptionkey@0"
I20260925 01:23:47.896442 14220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:47.915485 14220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:47.919623 14220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:47.919905 14220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:47.921072 14220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:47.922614 14220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:47.922722 14220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.922855 14220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:47.922933 14220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:47.951713 14220 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43397
I20260925 01:23:47.951776 14352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43397 every 8 connection(s)
I20260925 01:23:47.953037 14220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:47.956931 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14220
I20260925 01:23:47.957237 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13571
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299427, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
I20260925 01:23:47.992338 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.955731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:45545 (local address 127.6.236.189:35469)
0925 01:23:47.956731 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.956737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.957428 (+ 691us) server_negotiation.cc:407] Connection header received
0925 01:23:47.968395 (+ 10967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.968401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.968495 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.968574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.973620 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.974804 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.978385 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.978961 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.979060 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.984006 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.984011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.984016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.984141 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.987611 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.988621 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.988627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.988630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.988784 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.989357 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.989362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.989366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.989720 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.989888 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.991743 (+ 1855us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.991977 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":785,"threads_started":1}
Sep 25 01:23:47 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.131@KRBTEST.COM
I20260925 01:23:47.994612 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.956341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.6.236.130:45545)
0925 01:23:47.957276 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.957321 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.966079 (+ 8758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.969134 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.969146 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.969899 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.972492 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.972519 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.975091 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.975097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.977179 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.977194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.979392 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.979399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.979696 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.981186 (+ 1490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.981215 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.983756 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.987752 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.987761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.987780 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.988473 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.988948 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.988956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.988959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.989214 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.989863 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.989871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.990349 (+ 478us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.992547 (+ 2198us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.993100 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":418,"threads_started":1}
I20260925 01:23:47.996275 14355 heartbeater.cc:347] Connected to a master server at 127.6.236.189:35469
I20260925 01:23:47.996567 14355 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:47.997310 14355 heartbeater.cc:511] Master 127.6.236.189:35469 requested a full tablet report, sending...
I20260925 01:23:47.997898 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.957842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.6.236.130:49063)
0925 01:23:47.980393 (+ 22551us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.980412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.980536 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.980944 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.980949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.982549 (+ 1600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.982843 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.982855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.984397 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.984402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.986022 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.986038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.987043 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.987048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.987165 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.987655 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.987678 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.990789 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.994525 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.994530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.994534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.995573 (+ 1039us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.996001 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.996006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.996009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.996174 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.996888 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.996893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.997043 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.997186 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 01:23:47.997489 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22436,"mutex_wait_us":47,"thread_start_us":22127,"threads_started":1}
I20260925 01:23:47.999073 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.958170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:49063 (local address 127.6.236.188:36491)
0925 01:23:47.978433 (+ 20263us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.978439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.980642 (+ 2203us) server_negotiation.cc:407] Connection header received
0925 01:23:47.980710 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.980715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.980789 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.980896 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.983014 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.984174 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.986162 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.986854 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.986988 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.991023 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.991029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.991034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.991078 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.994359 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.995718 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.995723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.995727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.995864 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.996311 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.996316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.996319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.996675 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.996821 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:47.998497 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0925 01:23:47.998707 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20148,"thread_start_us":20271,"threads_started":1}
I20260925 01:23:47.999352 13841 ts_manager.cc:194] Registered new tserver with Master: 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:47.999910 14353 heartbeater.cc:347] Connected to a master server at 127.6.236.188:36491
I20260925 01:23:48.000062 14353 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:48.000473 14353 heartbeater.cc:511] Master 127.6.236.188:36491 requested a full tablet report, sending...
I20260925 01:23:48.000684 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.956002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:47557 (local address 127.6.236.190:43315)
0925 01:23:47.956437 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.956443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.979062 (+ 22619us) server_negotiation.cc:407] Connection header received
0925 01:23:47.979203 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.979209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.979304 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.979419 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.981268 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.982382 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.985785 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.986352 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.986473 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.991025 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.991029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.991033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.991065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.994900 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.995929 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.995935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.995938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.996097 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:47.998429 (+ 2332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:47.998434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:47.998437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:47.998742 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:47.998833 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.000107 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.000330 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260925 01:23:48.001238 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.956341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:47557)
0925 01:23:47.978947 (+ 22606us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:47.978966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:47.979088 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:47.980198 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:47.980203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:47.980815 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:47.981113 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.981126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.983776 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.983781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.985671 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:47.985685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.986517 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.986523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:47.986666 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.987443 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:47.987466 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:47.990827 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:47.995075 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.995080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.995084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.995805 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.998128 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:47.998132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:47.998135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:47.998299 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:47.999685 (+ 1386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:47.999690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:47.999831 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 01:23:47.999971 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.000785 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22454,"mutex_wait_us":315,"thread_start_us":23610,"threads_started":1}
I20260925 01:23:48.001577 14031 ts_manager.cc:194] Registered new tserver with Master: 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:48.002069 14354 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:48.002185 14354 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:48.002578 14354 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
I20260925 01:23:48.003434 13243 ts_manager.cc:194] Re-registered known tserver with Master: 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:48.003492 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:47.958639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.6.236.131:33891)
0925 01:23:47.978092 (+ 19453us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:47.978102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:47.979747 (+ 1645us) server_negotiation.cc:407] Connection header received
0925 01:23:47.980039 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:47.980048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:47.980281 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:47.980452 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:47.982518 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.983591 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.987781 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:47.988646 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:47.988760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:47.995480 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:47.995495 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:47.995513 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:47.995556 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:47.999246 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.001105 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.001114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.001118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.001272 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.001856 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.001860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.001863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.002289 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.002601 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.002884 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.003130 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19204,"thread_start_us":19456,"threads_started":1}
I20260925 01:23:48.004354 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:47557
I20260925 01:23:48.005870 13694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:48.096419 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13571
I20260925 01:23:48.111043 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:33891
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:43315
--webserver_port=38089
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:48.422032 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:48.422322 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:48.422463 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:48.429695 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:48.429795 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:48.429935 14366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:48.439956 14366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:33891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=38089
--tserver_master_addrs=127.6.236.190:43315,127.6.236.189:35469,127.6.236.188:36491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:48.441392 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:48.442675 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:23:48.443168 14210 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
W20260925 01:23:48.455636 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:48.455711 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:48.455669 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:48.456399 14366 server_base.cc:1034] running on GCE node
I20260925 01:23:48.456925 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:48.457772 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:48.458954 14366 hybrid_clock.cc:648] HybridClock initialized: now 1790299428458901 us; error 43 us; skew 500 ppm
I20260925 01:23:48.459249 14366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:48 dist-test-slave-s0b2 krb5kdc[13198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299428, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:48.462216 14366 init.cc:377] Logged in from keytab as oryx/127.6.236.131@KRBTEST.COM (short username oryx)
I20260925 01:23:48.463575 14366 webserver.cc:460] Webserver started at http://127.6.236.131:38089/ using document root <none> and password file <none>
I20260925 01:23:48.464211 14366 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:48.464332 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:48.469411 14366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:23:48.471971 14381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:48.473380 14366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:23:48.473526 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "bcd5bc3f63764e8198a3441bd27b1023"
format_stamp: "Formatted at 2026-09-25 01:23:45 on dist-test-slave-s0b2"
server_key: "0a4382ac7c81ca9ceb662773a624e9cb"
server_key_iv: "7a7efe9bcee856e7fc4a25e6e4d179ca"
server_key_version: "encryptionkey@0"
I20260925 01:23:48.474503 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:48.515223 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:48.519138 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:48.519379 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:48.520278 14366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:23:48.521562 14366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:48.521653 14366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:48.521756 14366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:48.521822 14366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:48.551662 14366 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:33891
I20260925 01:23:48.551723 14493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:33891 every 8 connection(s)
I20260925 01:23:48.553010 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:48.559931 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14366
Sep 25 01:23:48 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299428, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.188@KRBTEST.COM
Sep 25 01:23:48 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299428, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
Sep 25 01:23:48 dist-test-slave-s0b2 krb5kdc[13198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299428, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.189@KRBTEST.COM
I20260925 01:23:48.586364 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.558339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:41545 (local address 127.6.236.188:36491)
0925 01:23:48.561532 (+ 3193us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.561538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.561557 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:48.561613 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.561618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.561690 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.561793 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.564537 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.565591 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.567912 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.568576 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.568700 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.575540 (+ 6840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.575548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.575554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.575599 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.580171 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.581126 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.581132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.581136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.581295 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.581838 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.581843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.581847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.582206 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.582377 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.585663 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.585917 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3068,"thread_start_us":2959,"threads_started":1}
I20260925 01:23:48.587663 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.558144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.6.236.131:41545)
0925 01:23:48.558665 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.558677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.560491 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.561964 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.561968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.562555 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:48.563041 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.563068 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.565752 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.565758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.567744 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.567757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.569419 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.569425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.569597 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.570173 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:48.570195 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:48.575284 (+ 5089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:48.580343 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.580352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.580368 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.580975 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.581437 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.581442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.581445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.581707 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.582450 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:48.582457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:48.582901 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.583250 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.583803 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":26,"thread_start_us":279,"threads_started":1}
I20260925 01:23:48.589584 14494 heartbeater.cc:347] Connected to a master server at 127.6.236.188:36491
I20260925 01:23:48.589910 14494 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:48.590128 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.555865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:37937 (local address 127.6.236.189:35469)
0925 01:23:48.556948 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.556955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.557667 (+ 712us) server_negotiation.cc:407] Connection header received
0925 01:23:48.560398 (+ 2731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.560404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.560498 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.560610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.563247 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.564336 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.569377 (+ 5041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.570441 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.570575 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.578760 (+ 8185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.578767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.578772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.578830 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.582793 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.583550 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.583555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.583559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.583736 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.584117 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.584121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.584124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.584487 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.584640 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.589543 (+ 4903us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.589767 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":853,"threads_started":1}
I20260925 01:23:48.590840 14494 heartbeater.cc:511] Master 127.6.236.188:36491 requested a full tablet report, sending...
I20260925 01:23:48.591777 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.556564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.6.236.131:37937)
0925 01:23:48.557521 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.557559 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.560109 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.560975 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.560986 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.562038 (+ 1052us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:48.563041 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.563068 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.564480 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.564486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.569244 (+ 4758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.569256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.571798 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.571805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.571984 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.572592 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:48.572615 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:48.578395 (+ 5780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:48.582914 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.582918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.582922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.583435 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.583842 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.583847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.583851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.584012 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.589079 (+ 5067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:48.589085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:48.589274 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.589414 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.591289 (+ 1875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":259,"spinlock_wait_cycles":896,"thread_start_us":448,"threads_started":1}
I20260925 01:23:48.592725 14031 ts_manager.cc:194] Registered new tserver with Master: bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891)
I20260925 01:23:48.594009 14496 heartbeater.cc:347] Connected to a master server at 127.6.236.189:35469
I20260925 01:23:48.594153 14496 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:48.594636 14496 heartbeater.cc:511] Master 127.6.236.189:35469 requested a full tablet report, sending...
I20260925 01:23:48.594715 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.556564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:51339)
0925 01:23:48.557523 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.557560 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.560112 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.560974 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.560984 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.562047 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:48.563044 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.563069 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.564619 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.564624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.566830 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.566841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.567668 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.567676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.567946 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.569085 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:48.569122 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:48.576075 (+ 6953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:48.586762 (+ 10687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.586768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.586772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.587279 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.588085 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:48.588090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:48.588094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:48.588263 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:48.593672 (+ 5409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:48.593676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:48.593823 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.593957 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.594232 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":735,"spinlock_wait_cycles":35584,"thread_start_us":829,"threads_started":1}
I20260925 01:23:48.595422 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.555855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:51339 (local address 127.6.236.190:43315)
0925 01:23:48.557147 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.557153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.557718 (+ 565us) server_negotiation.cc:407] Connection header received
0925 01:23:48.560517 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.560523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.560600 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.560699 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.563209 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.564347 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.566963 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.567384 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.567492 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.576256 (+ 8764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.576260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.576264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.576306 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.582478 (+ 6172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.587399 (+ 4921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.587405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.587408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.587572 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.588368 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.588373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.588377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.588667 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.588806 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.594852 (+ 6046us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.595103 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1168,"thread_start_us":1069,"threads_started":1}
I20260925 01:23:48.597263 13841 ts_manager.cc:194] Registered new tserver with Master: bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891)
I20260925 01:23:48.598191 14495 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43315
I20260925 01:23:48.598362 14495 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:48.598760 14495 heartbeater.cc:511] Master 127.6.236.190:43315 requested a full tablet report, sending...
I20260925 01:23:48.599817 13243 ts_manager.cc:194] Re-registered known tserver with Master: bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891)
I20260925 01:23:48.600889 13243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:51339
I20260925 01:23:48.613358 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.567846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.6.236.190:43315)
0925 01:23:48.595644 (+ 27798us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.595650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.595667 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:48.595710 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.595713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.595766 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.595847 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.598072 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.599121 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.601288 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.601874 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.601988 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.607326 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.607333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.607338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.607375 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.610631 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.611480 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.611485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.611488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.611624 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.612021 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.612026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.612029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.612286 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.612502 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.612836 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.613033 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27706,"thread_start_us":38415,"threads_started":1}
I20260925 01:23:48.614960 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.574488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.6.236.189:35469)
0925 01:23:48.590447 (+ 15959us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.590453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.590469 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:48.590512 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.590517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.590594 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.590687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.592966 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.594595 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.603547 (+ 8952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.604232 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.604354 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.607753 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.607759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.607764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.607800 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.610705 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.611833 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.611839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.611843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.611993 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.612509 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.612514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.612529 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.612889 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.613108 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.614345 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.614591 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15862,"thread_start_us":31961,"threads_started":1}
I20260925 01:23:48.618957 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.584484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.6.236.188:36491)
0925 01:23:48.586690 (+ 2206us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.586697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.603604 (+ 16907us) server_negotiation.cc:407] Connection header received
0925 01:23:48.603822 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.603828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.603913 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.603978 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.606630 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.607608 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.609466 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.610064 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.610154 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.613620 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:48.613626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:48.613630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:48.613668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:48.616593 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.617175 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.617180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.617184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.617308 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:48.617641 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:48.617644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:48.617646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:48.617907 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:48.618113 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.618463 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.618685 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2102,"thread_start_us":18485,"threads_started":1}
I20260925 01:23:48.617571 13243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:48.624609 13243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:48.674443 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.640632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43397 (local address 127.0.0.1:48326)
0925 01:23:48.664958 (+ 24326us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.664981 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.665151 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.666669 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.666674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.666712 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:48.667168 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.667183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.669237 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.669243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.671641 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.671655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.673567 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.673573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.673739 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.673771 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.673861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.673989 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24173,"mutex_wait_us":23,"thread_start_us":26028,"threads_started":1}
I20260925 01:23:48.674970 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.641266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.6.236.131:33891)
0925 01:23:48.666319 (+ 25053us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.666327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.667061 (+ 734us) server_negotiation.cc:407] Connection header received
0925 01:23:48.667965 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.667973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.668208 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.668358 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:48.669216 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.670292 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.672775 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.673900 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.674011 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.674215 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.674516 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.674675 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24873,"thread_start_us":25063,"threads_started":1}
I20260925 01:23:48.675202 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.640845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.6.236.130:43397)
0925 01:23:48.665916 (+ 25071us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.665925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.665958 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:23:48.666067 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.666076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.666405 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.667405 (+ 1000us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:48.667628 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.669103 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.671774 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.673052 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.673207 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.673425 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.674349 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.674798 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24922,"thread_start_us":24810,"threads_started":1}
I20260925 01:23:48.675711 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.641016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:33891 (local address 127.0.0.1:49212)
0925 01:23:48.666921 (+ 25905us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.666939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.667800 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.668667 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.668672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.668696 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:48.669009 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.669018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.670465 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.670470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.672646 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.672658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.674103 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.674110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.674322 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.674354 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.675241 (+ 887us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.675370 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25772,"mutex_wait_us":32,"thread_start_us":26245,"threads_started":1}
I20260925 01:23:48.676066 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.641851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.6.236.129:42755)
0925 01:23:48.664218 (+ 22367us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.664229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.667651 (+ 3422us) server_negotiation.cc:407] Connection header received
0925 01:23:48.667804 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.667812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.668059 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.668250 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:48.668765 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.670056 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.672610 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.674149 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.674294 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.674491 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.675609 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.675773 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22159,"thread_start_us":22395,"threads_started":1}
I20260925 01:23:48.676415 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.641367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42755 (local address 127.0.0.1:36274)
0925 01:23:48.667520 (+ 26153us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.667536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.667665 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.668252 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.668256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.668280 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:48.668586 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.668595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.670202 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.670208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.672455 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.672470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.674628 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.674634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.674789 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.674818 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.674912 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.675042 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26045,"mutex_wait_us":1,"thread_start_us":23881,"threads_started":1}
I20260925 01:23:48.678776 14288 tablet_service.cc:1467] Processing CreateTablet for tablet f848154993c14cbdb15cc8beeacffbd2 (DEFAULT_TABLE table=test-table [id=4742c775c71943c19eb223ec5c40debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:48.678864 14429 tablet_service.cc:1467] Processing CreateTablet for tablet f848154993c14cbdb15cc8beeacffbd2 (DEFAULT_TABLE table=test-table [id=4742c775c71943c19eb223ec5c40debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:48.680114 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f848154993c14cbdb15cc8beeacffbd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:48.680537 14429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f848154993c14cbdb15cc8beeacffbd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:48.681562 14144 tablet_service.cc:1467] Processing CreateTablet for tablet f848154993c14cbdb15cc8beeacffbd2 (DEFAULT_TABLE table=test-table [id=4742c775c71943c19eb223ec5c40debd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:48.682847 14144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f848154993c14cbdb15cc8beeacffbd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:48.691299 14527 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Bootstrap starting.
I20260925 01:23:48.694492 14527 tablet_bootstrap.cc:654] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:48.695429 14527 log.cc:824] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:48.696666 14528 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Bootstrap starting.
I20260925 01:23:48.697886 14530 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Bootstrap starting.
I20260925 01:23:48.698359 14527 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: No bootstrap required, opened a new log
I20260925 01:23:48.698623 14527 ts_tablet_manager.cc:1397] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 01:23:48.700657 14528 tablet_bootstrap.cc:654] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:48.700706 14530 tablet_bootstrap.cc:654] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:48.701781 14530 log.cc:824] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:48.701807 14528 log.cc:824] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:48.709805 14527 raft_consensus.cc:348] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.710294 14527 raft_consensus.cc:374] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:48.710404 14527 raft_consensus.cc:729] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e99fa0e0a040878a25b6de75fc6871, State: Initialized, Role: FOLLOWER
I20260925 01:23:48.711017 14528 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: No bootstrap required, opened a new log
I20260925 01:23:48.711300 14528 ts_tablet_manager.cc:1397] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260925 01:23:48.711143 14527 consensus_queue.cc:260] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.711342 14530 tablet_bootstrap.cc:492] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: No bootstrap required, opened a new log
I20260925 01:23:48.711603 14530 ts_tablet_manager.cc:1397] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260925 01:23:48.719097 14354 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
I20260925 01:23:48.719573 14528 raft_consensus.cc:348] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.720028 14528 raft_consensus.cc:374] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:48.720120 14528 raft_consensus.cc:729] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd5bc3f63764e8198a3441bd27b1023, State: Initialized, Role: FOLLOWER
I20260925 01:23:48.720829 14528 consensus_queue.cc:260] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.722258 14530 raft_consensus.cc:348] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.722694 14530 raft_consensus.cc:374] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:48.722779 14530 raft_consensus.cc:729] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dbb7a5a84348b69ce6f413fab4c277, State: Initialized, Role: FOLLOWER
I20260925 01:23:48.723453 14530 consensus_queue.cc:260] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.724323 14527 ts_tablet_manager.cc:1428] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Time spent starting tablet: real 0.026s user 0.017s sys 0.004s
I20260925 01:23:48.725113 14530 ts_tablet_manager.cc:1428] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 01:23:48.728477 14495 heartbeater.cc:503] Master 127.6.236.190:43315 was elected leader, sending a full tablet report...
I20260925 01:23:48.728526 14528 ts_tablet_manager.cc:1428] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260925 01:23:48.805908 14497 tablet.cc:2369] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:48.859213 14534 raft_consensus.cc:482] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:48.859505 14534 raft_consensus.cc:504] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.861322 14534 leader_election.cc:288] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397), e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:48.867025 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.861641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36407 (local address 127.6.236.130:43397)
0925 01:23:48.861847 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.861854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.861873 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:48.862100 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.862107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.862207 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.862308 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:48.862726 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.863543 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.865299 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.866051 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.866189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.866283 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.866673 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.866797 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:23:48.867053 14526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.862149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50261 (local address 127.6.236.129:42755)
0925 01:23:48.862376 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.862382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.862400 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:23:48.862473 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.862477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.862546 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.862640 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:48.863003 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.863717 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.865333 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.866110 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.866282 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.866356 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.866674 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.866805 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:23:48.867084 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.861586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42755 (local address 127.6.236.131:50261)
0925 01:23:48.861834 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.861851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.861966 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.862669 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.862674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.862708 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:48.862879 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.862889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.863840 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.863844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.865180 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.865192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.866299 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.866306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.866469 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.866491 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.866561 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.866700 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":28,"spinlock_wait_cycles":1408}
I20260925 01:23:48.867110 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.861525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43397 (local address 127.6.236.131:36407)
0925 01:23:48.861808 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:48.861823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:48.861945 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:48.862347 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:48.862351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:48.862378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:48.862599 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.862611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.863674 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.863680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.865142 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:48.865154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.866205 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.866211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:48.866437 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.866470 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:23:48.866660 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 01:23:48.866797 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":21}
I20260925 01:23:48.868084 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f848154993c14cbdb15cc8beeacffbd2" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e99fa0e0a040878a25b6de75fc6871" is_pre_election: true
I20260925 01:23:48.868126 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f848154993c14cbdb15cc8beeacffbd2" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" is_pre_election: true
I20260925 01:23:48.868541 14308 raft_consensus.cc:2454] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 0.
I20260925 01:23:48.868541 14164 raft_consensus.cc:2454] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 0.
I20260925 01:23:48.869127 14383 leader_election.cc:302] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcd5bc3f63764e8198a3441bd27b1023, e1dbb7a5a84348b69ce6f413fab4c277; no voters:
I20260925 01:23:48.869472 14534 raft_consensus.cc:2781] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:48.869623 14534 raft_consensus.cc:482] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:48.869725 14534 raft_consensus.cc:3037] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:48.871848 14534 raft_consensus.cc:504] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.872432 14534 leader_election.cc:288] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 election: Requested vote from peers 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397), e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755)
I20260925 01:23:48.872795 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f848154993c14cbdb15cc8beeacffbd2" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e99fa0e0a040878a25b6de75fc6871"
I20260925 01:23:48.872931 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f848154993c14cbdb15cc8beeacffbd2" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
I20260925 01:23:48.872989 14308 raft_consensus.cc:3037] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:48.873126 14164 raft_consensus.cc:3037] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:48.875321 14308 raft_consensus.cc:2454] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 1.
I20260925 01:23:48.875566 14164 raft_consensus.cc:2454] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 1.
I20260925 01:23:48.875690 14382 leader_election.cc:302] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e99fa0e0a040878a25b6de75fc6871, bcd5bc3f63764e8198a3441bd27b1023; no voters:
I20260925 01:23:48.875981 14534 raft_consensus.cc:2781] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:48.876367 14534 raft_consensus.cc:686] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 LEADER]: Becoming Leader. State: Replica: bcd5bc3f63764e8198a3441bd27b1023, State: Running, Role: LEADER
I20260925 01:23:48.877038 14534 consensus_queue.cc:237] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } }
I20260925 01:23:48.881959 13241 catalog_manager.cc:5495] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "bcd5bc3f63764e8198a3441bd27b1023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } health_report { overall_health: UNKNOWN } } }
W20260925 01:23:48.908002 14216 tablet.cc:2369] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:48.912624 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.907312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.6.236.131:33891)
0925 01:23:48.907505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.907511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.907612 (+ 101us) server_negotiation.cc:407] Connection header received
0925 01:23:48.907868 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.907872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.907953 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.908083 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:48.908501 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.909262 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.910708 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.911096 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.911182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:48.911614 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:48.912108 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:48.912219 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:48.912305 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 01:23:48.912370 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:48.920454 14164 raft_consensus.cc:1261] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Refusing update from remote peer bcd5bc3f63764e8198a3441bd27b1023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:48.920512 14308 raft_consensus.cc:1261] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Refusing update from remote peer bcd5bc3f63764e8198a3441bd27b1023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:48.921397 14536 consensus_queue.cc:1035] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:48.921789 14534 consensus_queue.cc:1035] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:48.939037 14542 mvcc.cc:205] Tried to move back new op lower bound from 7333066460848766976 to 7333066460683726848. Current Snapshot: MvccSnapshot[applied={T|T < 7333066460848766976}]
I20260925 01:23:48.953965 13241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:48.954272 13241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:48.961057 13241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38b03a55932f4ed2a155f9c851b60407: Sending DeleteTablet for 3 replicas of tablet f848154993c14cbdb15cc8beeacffbd2
I20260925 01:23:48.962533 14429 tablet_service.cc:1514] Processing DeleteTablet for tablet f848154993c14cbdb15cc8beeacffbd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:48 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:49212
I20260925 01:23:48.962927 14144 tablet_service.cc:1514] Processing DeleteTablet for tablet f848154993c14cbdb15cc8beeacffbd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:48 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:36274
I20260925 01:23:48.963824 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet f848154993c14cbdb15cc8beeacffbd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:48 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:48326
I20260925 01:23:48.981804 14550 tablet_replica.cc:330] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: stopping tablet replica
I20260925 01:23:48.982985 14550 raft_consensus.cc:2229] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:48.984227 14552 tablet_replica.cc:330] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: stopping tablet replica
I20260925 01:23:48.984756 14552 raft_consensus.cc:2229] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:48.985541 14550 raft_consensus.cc:2258] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:48.986920 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13211
I20260925 01:23:48.987730 14552 raft_consensus.cc:2258] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:48.987962 13273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:48.988547 14551 tablet_replica.cc:330] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: stopping tablet replica
I20260925 01:23:48.989075 14551 raft_consensus.cc:2229] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:48.990598 14552 ts_tablet_manager.cc:1905] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:48.995739 14550 ts_tablet_manager.cc:1905] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:48.997699 14552 ts_tablet_manager.cc:1918] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:48.997874 14552 log.cc:1196] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/f848154993c14cbdb15cc8beeacffbd2
I20260925 01:23:48.998755 14551 raft_consensus.cc:2258] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:49.001241 14551 ts_tablet_manager.cc:1905] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:49.005013 14552 ts_tablet_manager.cc:1939] T f848154993c14cbdb15cc8beeacffbd2 P 08e99fa0e0a040878a25b6de75fc6871: Deleting consensus metadata
I20260925 01:23:49.010432 14551 ts_tablet_manager.cc:1918] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:49.010643 14551 log.cc:1196] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/f848154993c14cbdb15cc8beeacffbd2
I20260925 01:23:49.011204 14551 ts_tablet_manager.cc:1939] T f848154993c14cbdb15cc8beeacffbd2 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting consensus metadata
I20260925 01:23:49.015859 14550 ts_tablet_manager.cc:1918] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:49.016901 14550 log.cc:1196] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/f848154993c14cbdb15cc8beeacffbd2
I20260925 01:23:49.017601 14550 ts_tablet_manager.cc:1939] T f848154993c14cbdb15cc8beeacffbd2 P bcd5bc3f63764e8198a3441bd27b1023: Deleting consensus metadata
I20260925 01:23:49.021782 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.970778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.6.236.189:35469)
0925 01:23:48.970962 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.970968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:48.975089 (+ 4121us) server_negotiation.cc:407] Connection header received
0925 01:23:48.978857 (+ 3768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:48.978863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:48.978946 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:48.979014 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:48.986334 (+ 7320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.987685 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.999049 (+ 11364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:48.999717 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:48.999836 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:49.008215 (+ 8379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:49.008223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:49.008228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:49.008278 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:49.018889 (+ 10611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.019810 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.019815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.019817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.019948 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.020329 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.020334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.020336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.020629 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:49.020857 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:49.021206 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 01:23:49.021413 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:23:49.038815 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:48.975192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.6.236.188:36491)
0925 01:23:48.975363 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:48.975369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:49.022493 (+ 47124us) server_negotiation.cc:407] Connection header received
0925 01:23:49.022745 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:49.022751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:49.022831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:49.022912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:49.024814 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.025964 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.027853 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.028505 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.028596 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:49.032029 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:49.032035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:49.032040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:49.032102 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:49.035632 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.036390 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.036399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.036403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.036574 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.037004 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.037009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.037013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.037373 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:49.037622 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:49.038147 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 01:23:49.038402 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:23:49.157810 13228 catalog_manager.cc:4841] TS e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755): tablet f848154993c14cbdb15cc8beeacffbd2 (table test-table [id=4742c775c71943c19eb223ec5c40debd]) successfully deleted
I20260925 01:23:49.159873 13227 catalog_manager.cc:4841] TS 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397): tablet f848154993c14cbdb15cc8beeacffbd2 (table test-table [id=4742c775c71943c19eb223ec5c40debd]) successfully deleted
I20260925 01:23:49.160218 13230 catalog_manager.cc:4841] TS bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891): tablet f848154993c14cbdb15cc8beeacffbd2 (table test-table [id=4742c775c71943c19eb223ec5c40debd]) successfully deleted
W20260925 01:23:49.160102 13227 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.236.130:43397, user_credentials={real_user=slave}} blocked reactor thread for 150514us
I20260925 01:23:49.160794 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13211
I20260925 01:23:49.200938 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:49.182862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.6.236.189:35469)
0925 01:23:49.183089 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:49.183118 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:49.184063 (+ 945us) server_negotiation.cc:407] Connection header received
0925 01:23:49.184351 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:49.184357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:49.184441 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:49.184510 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:49.187099 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.188498 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.190983 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.191762 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.191856 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:49.195271 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:49.195276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:49.195279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:49.195313 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:49.197828 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.198481 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.198485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.198487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.198604 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.199343 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.199347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.199349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.199788 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:49.200010 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:49.200391 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 01:23:49.200596 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:23:49.201687 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:49.184126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.6.236.188:36491)
0925 01:23:49.184329 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:49.184336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:49.184356 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:23:49.184503 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:49.184509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:49.184596 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:49.184694 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:49.186983 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.188105 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.190011 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:49.190650 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:49.190751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:49.195240 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:49.195247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:49.195251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:49.195293 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:49.198554 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.199148 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.199153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.199157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.199307 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:49.199731 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:49.199736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:49.199739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:49.200090 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:49.200328 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:49.201153 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 01:23:49.201361 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:23:49.945588 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:49.944892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:58747)
0925 01:23:49.945318 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:49.945433 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":224,"threads_started":1}
W20260925 01:23:49.946010 14210 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:43315 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:49.949652 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:49.948508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:38713)
0925 01:23:49.949186 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:49.949369 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":427,"threads_started":1}
I20260925 01:23:49.950109 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:49.949313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:56913)
0925 01:23:49.949807 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:49.949937 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":284,"threads_started":1}
W20260925 01:23:49.950201 14495 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:43315 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
W20260925 01:23:49.950521 14354 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:43315 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.461870 14564 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 38b03a55932f4ed2a155f9c851b60407)
I20260925 01:23:50.462132 14564 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } }
I20260925 01:23:50.462764 14565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 38b03a55932f4ed2a155f9c851b60407)
I20260925 01:23:50.462994 14565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } }
I20260925 01:23:50.463583 14564 leader_election.cc:288] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315), 514d558695924e5fb2a9071df8919ebf (127.6.236.189:35469)
I20260925 01:23:50.464342 14565 leader_election.cc:288] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315), b4f8299c682c4c3891d02156d15c5937 (127.6.236.188:36491)
I20260925 01:23:50.464766 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.463709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57762)
0925 01:23:50.464299 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.464480 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":317,"threads_started":1}
W20260925 01:23:50.465679 14015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.466552 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.464408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57764)
0925 01:23:50.466062 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.466279 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1535,"thread_start_us":1741,"threads_started":1}
W20260925 01:23:50.466851 13825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.474064 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.464503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:36491 (local address 127.0.0.1:33612)
0925 01:23:50.466799 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.466815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:50.466944 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:50.467463 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:50.467468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:50.467501 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:50.467763 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.467774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.469388 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.469394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.471650 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.471667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.473328 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.473335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.473524 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.473559 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:50.473659 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:23:50.473797 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2201,"threads_started":1}
I20260925 01:23:50.474772 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.464886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.6.236.188:36491)
0925 01:23:50.465681 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.465686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.467049 (+ 1363us) server_negotiation.cc:407] Connection header received
0925 01:23:50.467121 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.467127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.467207 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.467286 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:50.467934 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.469218 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.471854 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.473010 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.473119 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.473220 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.474262 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.474398 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":526,"threads_started":1}
I20260925 01:23:50.476701 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.476248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57774)
0925 01:23:50.476454 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.476535 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260925 01:23:50.477049 13825 leader_election.cc:334] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315): Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.478215 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.477462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57784)
0925 01:23:50.477726 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.477948 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260925 01:23:50.478555 14015 leader_election.cc:334] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315): Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.478508 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "514d558695924e5fb2a9071df8919ebf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b4f8299c682c4c3891d02156d15c5937" is_pre_election: true
I20260925 01:23:50.478853 14041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 514d558695924e5fb2a9071df8919ebf in term 1.
I20260925 01:23:50.479301 13826 leader_election.cc:302] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 514d558695924e5fb2a9071df8919ebf, b4f8299c682c4c3891d02156d15c5937; no voters: 38b03a55932f4ed2a155f9c851b60407
I20260925 01:23:50.479743 14565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 01:23:50.479890 14565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 38b03a55932f4ed2a155f9c851b60407)
I20260925 01:23:50.479991 14565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:23:50.482733 14565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } }
I20260925 01:23:50.483323 14565 leader_election.cc:288] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 election: Requested vote from peers 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315), b4f8299c682c4c3891d02156d15c5937 (127.6.236.188:36491)
I20260925 01:23:50.483737 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "514d558695924e5fb2a9071df8919ebf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b4f8299c682c4c3891d02156d15c5937"
I20260925 01:23:50.483862 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.463788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:35469 (local address 127.0.0.1:48202)
0925 01:23:50.465054 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.465072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:50.465206 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:50.477802 (+ 12596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:50.477807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:50.477837 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:50.478133 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.478149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.479776 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.479782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.481747 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.481762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.483080 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.483086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.483304 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.483339 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:50.483437 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:23:50.483577 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":3887,"threads_started":1}
I20260925 01:23:50.483968 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.483446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57800)
0925 01:23:50.483609 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.483772 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 01:23:50.483950 14041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:23:50.484660 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.464650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.6.236.189:35469)
0925 01:23:50.477482 (+ 12832us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.477488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.477508 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:23:50.477575 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.477579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.477649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.478301 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:50.478578 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.479604 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.481934 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.482894 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.483040 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.483150 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.484214 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.484345 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12650,"thread_start_us":12476,"threads_started":1}
I20260925 01:23:50.485088 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.484683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57812)
0925 01:23:50.484885 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.484964 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260925 01:23:50.485270 13825 leader_election.cc:334] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315): Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
I20260925 01:23:50.485266 13850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4f8299c682c4c3891d02156d15c5937" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "514d558695924e5fb2a9071df8919ebf" is_pre_election: true
I20260925 01:23:50.485637 13850 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b4f8299c682c4c3891d02156d15c5937 in current term 2: Already voted for candidate 514d558695924e5fb2a9071df8919ebf in this term.
I20260925 01:23:50.486203 14016 leader_election.cc:302] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4f8299c682c4c3891d02156d15c5937; no voters: 38b03a55932f4ed2a155f9c851b60407, 514d558695924e5fb2a9071df8919ebf
I20260925 01:23:50.486521 14041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 514d558695924e5fb2a9071df8919ebf in term 2.
I20260925 01:23:50.486672 14564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:23:50.486941 13826 leader_election.cc:302] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 514d558695924e5fb2a9071df8919ebf, b4f8299c682c4c3891d02156d15c5937; no voters: 38b03a55932f4ed2a155f9c851b60407
I20260925 01:23:50.487172 14565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:23:50.487599 14565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [term 2 LEADER]: Becoming Leader. State: Replica: 514d558695924e5fb2a9071df8919ebf, State: Running, Role: LEADER
I20260925 01:23:50.487938 14565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } }
I20260925 01:23:50.489404 14574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514d558695924e5fb2a9071df8919ebf. Latest consensus state: current_term: 2 leader_uuid: "514d558695924e5fb2a9071df8919ebf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:50.489699 14574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:50.490341 14576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:50.494051 14576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:50.494930 14576 catalog_manager.cc:1259] Loaded cluster ID: 6a364ac311484adf97fb375bc795e735
I20260925 01:23:50.495004 14576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:50.496071 14576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:50.496845 14576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Loaded TSK: 0
I20260925 01:23:50.497645 14576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:23:50.532248 13840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:50.538985 13840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:50.542934 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.542557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57828)
0925 01:23:50.542719 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.542799 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 01:23:50.543036 14041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [term 2 FOLLOWER]: Refusing update from remote peer 514d558695924e5fb2a9071df8919ebf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 01:23:50.543702 14574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 01:23:50.543867 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.543552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57832)
0925 01:23:50.543693 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.543758 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:23:50.544082 13825 consensus_peers.cc:487] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf -> Peer 38b03a55932f4ed2a155f9c851b60407 (127.6.236.190:43315): Couldn't send request to peer 38b03a55932f4ed2a155f9c851b60407. Status: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 01:23:50.546303 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514d558695924e5fb2a9071df8919ebf. Latest consensus state: current_term: 2 leader_uuid: "514d558695924e5fb2a9071df8919ebf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:50.546566 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:23:50.547924 14574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "514d558695924e5fb2a9071df8919ebf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:50.548208 14574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:50.548568 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "514d558695924e5fb2a9071df8919ebf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "38b03a55932f4ed2a155f9c851b60407" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43315 } } peers { permanent_uuid: "514d558695924e5fb2a9071df8919ebf" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b4f8299c682c4c3891d02156d15c5937" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 36491 } attrs { promote: false } } }
I20260925 01:23:50.548856 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f8299c682c4c3891d02156d15c5937 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:23:50.564373 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.556945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.6.236.131:33891)
0925 01:23:50.557467 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.557473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.557492 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:50.557558 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.557563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.557633 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.558511 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:50.558692 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.559691 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.562320 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.563490 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.563597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.563702 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.564046 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.564146 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":490,"threads_started":1}
I20260925 01:23:50.564484 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.556677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:33891 (local address 127.0.0.1:49228)
0925 01:23:50.556819 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.556838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:50.556936 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:50.557876 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:50.557882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:50.557914 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:50.558361 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.558374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.559827 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.559832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.562143 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.562161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.563677 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.563683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.563872 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.563915 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 01:23:50.564022 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 01:23:50.564163 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260925 01:23:50.565250 14429 tablet_service.cc:1467] Processing CreateTablet for tablet 579c430234424c47b0fabf06ecc84692 (DEFAULT_TABLE table=test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:50.565692 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.558336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.6.236.130:43397)
0925 01:23:50.558772 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.558778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.558794 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:23:50.558857 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.558861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.558929 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.559031 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:50.559601 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.560577 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.562297 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.563444 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.563597 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.563695 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.565263 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.565407 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":206,"threads_started":1}
I20260925 01:23:50.565730 14429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c430234424c47b0fabf06ecc84692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:50.566427 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.557418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43397 (local address 127.0.0.1:48338)
0925 01:23:50.558559 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.558575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:50.558677 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:50.559045 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:50.559050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:50.559072 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:50.559427 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.559438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.560678 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.560683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.562129 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.562141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.564879 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.564885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.565070 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.565106 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:23:50.565196 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 01:23:50.565330 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":41,"thread_start_us":819,"threads_started":1}
I20260925 01:23:50.567262 14288 tablet_service.cc:1467] Processing CreateTablet for tablet 579c430234424c47b0fabf06ecc84692 (DEFAULT_TABLE table=test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:50.567710 14288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c430234424c47b0fabf06ecc84692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:50.572434 14586 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Bootstrap starting.
I20260925 01:23:50.574108 14587 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Bootstrap starting.
I20260925 01:23:50.574321 14586 tablet_bootstrap.cc:654] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:50.575086 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.557397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42755 (local address 127.0.0.1:36290)
0925 01:23:50.557713 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.557731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:50.557819 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:50.569940 (+ 12121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:50.569945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:50.569973 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:50.570174 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.570186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.571433 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.571439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.572930 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:50.572946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.574393 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.574399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:50.574579 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.574610 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:23:50.574698 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 01:23:50.574833 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260925 01:23:50.575697 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.558020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.6.236.129:42755)
0925 01:23:50.569514 (+ 11494us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.569519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.569534 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:23:50.569592 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.569596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.569674 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.570339 (+ 665us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:50.570518 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.571300 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.573087 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.574226 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.574371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.574484 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.575286 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.575428 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11387,"thread_start_us":11727,"threads_started":1}
I20260925 01:23:50.575819 14587 tablet_bootstrap.cc:654] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:50.576421 14144 tablet_service.cc:1467] Processing CreateTablet for tablet 579c430234424c47b0fabf06ecc84692 (DEFAULT_TABLE table=test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:50.576834 14144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 579c430234424c47b0fabf06ecc84692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:50.577558 14586 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: No bootstrap required, opened a new log
I20260925 01:23:50.577680 14586 ts_tablet_manager.cc:1397] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260925 01:23:50.577883 14587 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: No bootstrap required, opened a new log
I20260925 01:23:50.578006 14587 ts_tablet_manager.cc:1397] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:23:50.578286 14586 raft_consensus.cc:348] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.578510 14586 raft_consensus.cc:374] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:50.578627 14586 raft_consensus.cc:729] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd5bc3f63764e8198a3441bd27b1023, State: Initialized, Role: FOLLOWER
I20260925 01:23:50.578856 14586 consensus_queue.cc:260] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.578826 14587 raft_consensus.cc:348] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.579114 14587 raft_consensus.cc:374] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:50.579221 14587 raft_consensus.cc:729] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e99fa0e0a040878a25b6de75fc6871, State: Initialized, Role: FOLLOWER
I20260925 01:23:50.579489 14587 consensus_queue.cc:260] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.579674 14586 ts_tablet_manager.cc:1428] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:23:50.585992 14496 heartbeater.cc:503] Master 127.6.236.189:35469 was elected leader, sending a full tablet report...
I20260925 01:23:50.586805 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.580597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:52717)
0925 01:23:50.586373 (+ 5776us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.586550 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5623,"threads_started":1}
I20260925 01:23:50.588418 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.588020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:42155)
0925 01:23:50.588213 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.588291 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 01:23:50.589176 14587 ts_tablet_manager.cc:1428] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260925 01:23:50.589329 14593 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Bootstrap starting.
I20260925 01:23:50.591012 14593 tablet_bootstrap.cc:654] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:50.592664 14355 heartbeater.cc:503] Master 127.6.236.189:35469 was elected leader, sending a full tablet report...
I20260925 01:23:50.593434 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.590900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:46195)
0925 01:23:50.593128 (+ 2228us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.593251 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2122,"thread_start_us":1989,"threads_started":1}
I20260925 01:23:50.594559 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.594075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:49253)
0925 01:23:50.594350 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.594429 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 01:23:50.594870 14593 tablet_bootstrap.cc:492] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: No bootstrap required, opened a new log
I20260925 01:23:50.594990 14593 ts_tablet_manager.cc:1397] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 01:23:50.595672 14593 raft_consensus.cc:348] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.595855 14593 raft_consensus.cc:374] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:50.595917 14593 raft_consensus.cc:729] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dbb7a5a84348b69ce6f413fab4c277, State: Initialized, Role: FOLLOWER
I20260925 01:23:50.596122 14593 consensus_queue.cc:260] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.598759 14593 ts_tablet_manager.cc:1428] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 01:23:50.599911 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.598979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:40879)
0925 01:23:50.599511 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.599666 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260925 01:23:50.600587 14211 heartbeater.cc:503] Master 127.6.236.189:35469 was elected leader, sending a full tablet report...
I20260925 01:23:50.601193 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.600784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:34687)
0925 01:23:50.600994 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.601073 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 01:23:50.708001 14356 tablet.cc:2369] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:50.722824 14590 raft_consensus.cc:482] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:50.723061 14590 raft_consensus.cc:504] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.723754 14590 leader_election.cc:288] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755), 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:50.724359 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c430234424c47b0fabf06ecc84692" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" is_pre_election: true
I20260925 01:23:50.724349 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c430234424c47b0fabf06ecc84692" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e99fa0e0a040878a25b6de75fc6871" is_pre_election: true
I20260925 01:23:50.724598 14164 raft_consensus.cc:2454] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 0.
I20260925 01:23:50.724624 14308 raft_consensus.cc:2454] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 0.
I20260925 01:23:50.725054 14383 leader_election.cc:302] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcd5bc3f63764e8198a3441bd27b1023, e1dbb7a5a84348b69ce6f413fab4c277; no voters:
I20260925 01:23:50.725337 14590 raft_consensus.cc:2781] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:50.725450 14590 raft_consensus.cc:482] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:50.725541 14590 raft_consensus.cc:3037] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:50.727620 14590 raft_consensus.cc:504] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.728122 14590 leader_election.cc:288] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 election: Requested vote from peers e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755), 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397)
I20260925 01:23:50.728672 14308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c430234424c47b0fabf06ecc84692" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e99fa0e0a040878a25b6de75fc6871"
I20260925 01:23:50.728688 14164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "579c430234424c47b0fabf06ecc84692" candidate_uuid: "bcd5bc3f63764e8198a3441bd27b1023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dbb7a5a84348b69ce6f413fab4c277"
I20260925 01:23:50.728886 14164 raft_consensus.cc:3037] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:50.728886 14308 raft_consensus.cc:3037] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:50.731711 14164 raft_consensus.cc:2454] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 1.
I20260925 01:23:50.731710 14308 raft_consensus.cc:2454] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd5bc3f63764e8198a3441bd27b1023 in term 1.
I20260925 01:23:50.732110 14382 leader_election.cc:302] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08e99fa0e0a040878a25b6de75fc6871, bcd5bc3f63764e8198a3441bd27b1023, e1dbb7a5a84348b69ce6f413fab4c277; no voters:
I20260925 01:23:50.732415 14590 raft_consensus.cc:2781] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:50.732821 14590 raft_consensus.cc:686] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 LEADER]: Becoming Leader. State: Replica: bcd5bc3f63764e8198a3441bd27b1023, State: Running, Role: LEADER
I20260925 01:23:50.733086 14590 consensus_queue.cc:237] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } }
I20260925 01:23:50.733902 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.733426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:48165)
0925 01:23:50.733603 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.733699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 01:23:50.735826 13840 catalog_manager.cc:5495] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "bcd5bc3f63764e8198a3441bd27b1023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcd5bc3f63764e8198a3441bd27b1023" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 33891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:50.756102 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.751803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.6.236.131:33891)
0925 01:23:50.751991 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.751997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.752016 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:50.752114 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.752120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.752192 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.752263 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:50.752607 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.753306 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.754600 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.755020 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.755126 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.755481 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:50.755579 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:50.755651 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.755789 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.755864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:23:50.759032 14308 raft_consensus.cc:1261] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Refusing update from remote peer bcd5bc3f63764e8198a3441bd27b1023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:50.759032 14164 raft_consensus.cc:1261] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Refusing update from remote peer bcd5bc3f63764e8198a3441bd27b1023: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:50.759610 14598 consensus_queue.cc:1035] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e99fa0e0a040878a25b6de75fc6871" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:50.759938 14590 consensus_queue.cc:1035] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dbb7a5a84348b69ce6f413fab4c277" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:50.762714 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.761775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:36127)
0925 01:23:50.762377 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.762497 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490}
I20260925 01:23:50.763154 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.761604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:43575)
0925 01:23:50.762030 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.762152 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
I20260925 01:23:50.768597 14602 mvcc.cc:205] Tried to move back new op lower bound from 7333066468383117312 to 7333066468285276160. Current Snapshot: MvccSnapshot[applied={T|T < 7333066468383117312}]
I20260925 01:23:50.770726 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.770310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:58287)
0925 01:23:50.770492 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.770588 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 01:23:50.770854 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.770481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.129:57265)
0925 01:23:50.770638 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.770729 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:23:50.775580 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.774190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.131:46051)
0925 01:23:50.775331 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.775418 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1041}
I20260925 01:23:50.776078 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.775600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.6.236.130:46775)
0925 01:23:50.775848 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:50.775931 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260925 01:23:50.777818 13840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:50.778124 13840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:50.784510 13840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 514d558695924e5fb2a9071df8919ebf: Sending DeleteTablet for 3 replicas of tablet 579c430234424c47b0fabf06ecc84692
I20260925 01:23:50.785663 14144 tablet_service.cc:1514] Processing DeleteTablet for tablet 579c430234424c47b0fabf06ecc84692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:50 UTC) from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:36290
I20260925 01:23:50.785840 14143 tablet_service.cc:1514] Processing DeleteTablet for tablet 579c430234424c47b0fabf06ecc84692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:50 UTC) from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:36290
I20260925 01:23:50.786139 14288 tablet_service.cc:1514] Processing DeleteTablet for tablet 579c430234424c47b0fabf06ecc84692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:50 UTC) from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:48338
I20260925 01:23:50.786361 14610 tablet_replica.cc:330] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: stopping tablet replica
I20260925 01:23:50.786538 14610 raft_consensus.cc:2229] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:50.786448 14287 tablet_service.cc:1514] Processing DeleteTablet for tablet 579c430234424c47b0fabf06ecc84692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:50 UTC) from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:48338
I20260925 01:23:50.786824 14610 raft_consensus.cc:2258] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:50.787137 14611 tablet_replica.cc:330] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: stopping tablet replica
I20260925 01:23:50.787165 14429 tablet_service.cc:1514] Processing DeleteTablet for tablet 579c430234424c47b0fabf06ecc84692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:50 UTC) from {username='oryx', principal='oryx/127.6.236.189@KRBTEST.COM'} at 127.0.0.1:49228
I20260925 01:23:50.787304 14611 raft_consensus.cc:2229] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:50.787570 14611 raft_consensus.cc:2258] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:50.787909 14610 ts_tablet_manager.cc:1905] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:50.788362 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14003
I20260925 01:23:50.791148 14611 ts_tablet_manager.cc:1905] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:50.794304 14610 ts_tablet_manager.cc:1918] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:50.794445 14610 log.cc:1196] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/579c430234424c47b0fabf06ecc84692
I20260925 01:23:50.794886 14610 ts_tablet_manager.cc:1939] T 579c430234424c47b0fabf06ecc84692 P e1dbb7a5a84348b69ce6f413fab4c277: Deleting consensus metadata
I20260925 01:23:50.796283 13826 catalog_manager.cc:4841] TS e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755): tablet 579c430234424c47b0fabf06ecc84692 (table test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]) successfully deleted
I20260925 01:23:50.797071 14611 ts_tablet_manager.cc:1918] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:50.797201 14611 log.cc:1196] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/579c430234424c47b0fabf06ecc84692
W20260925 01:23:50.797197 13826 catalog_manager.cc:4808] TS e1dbb7a5a84348b69ce6f413fab4c277 (127.6.236.129:42755): delete failed for tablet 579c430234424c47b0fabf06ecc84692 because the tablet was not found. No further retry: Not found: Tablet not found: 579c430234424c47b0fabf06ecc84692
I20260925 01:23:50.797592 14611 ts_tablet_manager.cc:1939] T 579c430234424c47b0fabf06ecc84692 P 08e99fa0e0a040878a25b6de75fc6871: Deleting consensus metadata
I20260925 01:23:50.798466 13825 catalog_manager.cc:4841] TS 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397): tablet 579c430234424c47b0fabf06ecc84692 (table test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]) successfully deleted
I20260925 01:23:50.798512 14612 tablet_replica.cc:330] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: stopping tablet replica
W20260925 01:23:50.798720 13825 catalog_manager.cc:4808] TS 08e99fa0e0a040878a25b6de75fc6871 (127.6.236.130:43397): delete failed for tablet 579c430234424c47b0fabf06ecc84692 because the tablet was not found. No further retry: Not found: Tablet not found: 579c430234424c47b0fabf06ecc84692
I20260925 01:23:50.798698 14612 raft_consensus.cc:2229] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:50.799098 14612 raft_consensus.cc:2258] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:50.800040 14612 ts_tablet_manager.cc:1905] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:50.804687 14612 ts_tablet_manager.cc:1918] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:50.804805 14612 log.cc:1196] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/579c430234424c47b0fabf06ecc84692
I20260925 01:23:50.805184 14612 ts_tablet_manager.cc:1939] T 579c430234424c47b0fabf06ecc84692 P bcd5bc3f63764e8198a3441bd27b1023: Deleting consensus metadata
I20260925 01:23:50.806085 13828 catalog_manager.cc:4841] TS bcd5bc3f63764e8198a3441bd27b1023 (127.6.236.131:33891): tablet 579c430234424c47b0fabf06ecc84692 (table test-table [id=88af3cb8241b477abe78c7c36ccbbe5a]) successfully deleted
I20260925 01:23:50.811120 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.790062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.6.236.129:42755)
0925 01:23:50.790278 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.790285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.799050 (+ 8765us) server_negotiation.cc:407] Connection header received
0925 01:23:50.799184 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.799189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.799267 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.799371 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:50.800743 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.801499 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.802766 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.803187 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.803290 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.805743 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:50.805758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:50.805773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:50.805807 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:50.808543 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:50.809207 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:50.809215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:50.809217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:50.809339 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:50.809662 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:50.809667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:50.809671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:50.810014 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:50.810336 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.810629 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.810836 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 01:23:50.811724 14204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:50.901963 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14003
I20260925 01:23:50.917665 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14220
I20260925 01:23:50.929816 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:50.918503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.6.236.130:43397)
0925 01:23:50.918745 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:50.918750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:50.918787 (+ 37us) server_negotiation.cc:407] Connection header received
0925 01:23:50.918841 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:50.918845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:50.918914 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:50.918985 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:50.920310 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.920997 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.922139 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:50.922583 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:50.922668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:50.924947 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:50.924963 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:50.924979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:50.925011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:50.927528 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:50.928156 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:50.928162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:50.928165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:50.928285 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:50.928581 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:50.928585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:50.928587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:50.928935 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:50.929143 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:50.929350 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:23:50.929504 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:23:50.930689 14348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:51.007783 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:51.007260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57844)
0925 01:23:51.007467 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:51.007570 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:23:51.008882 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:51.008503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43315 (local address 127.0.0.1:57860)
0925 01:23:51.008668 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:51.008745 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:23:51.023640 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14220
I20260925 01:23:51.044790 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14366
I20260925 01:23:51.058174 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:51.045516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.6.236.131:33891)
0925 01:23:51.045696 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:51.045702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:51.045722 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:23:51.045913 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:51.045917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:51.045999 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:51.046087 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:51.047509 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:51.048226 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:51.049479 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:51.049889 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:51.049997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:51.052723 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:51.052739 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:51.052756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:51.052794 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:51.055484 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:51.056143 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:51.056151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:51.056157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:51.056330 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:51.056674 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:51.056679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:51.056682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:51.057033 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:51.057331 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:51.057597 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 01:23:51.057803 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:23:51.058830 14489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:51.175045 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14366
I20260925 01:23:51.196002 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13704
I20260925 01:23:51.197057 13871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:51.317592 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13704
W20260925 01:23:51.334758 14015 connection.cc:535] server connection from 127.0.0.1:33612 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:23:51.335078 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 13805
I20260925 01:23:51.336063 14061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:51.448812 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 13805
2026-09-25T01:23:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8988 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:51 dist-test-slave-s0b2 krb5kdc[14621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:51 dist-test-slave-s0b2 krb5kdc[14621](info): set up 2 sockets
Sep 25 01:23:51 dist-test-slave-s0b2 krb5kdc[14621](info): commencing operation
krb5kdc: starting...
W20260925 01:23:52.451743 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.008s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:52 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab.
Entry for principal oryx/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab.
I20260925 01:23:52.643730 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43867
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab
--principal=oryx/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:43867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:52.953884 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:52.954171 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:52.954327 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:52.961526 14634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:52.961637 14634 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:23:52.961705 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:52.961787 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:52.961853 14634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:52.961931 14634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:52.971827 14634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:43867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.190.keytab
--principal=oryx/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:52.973127 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:52.974448 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:52.985579 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:52.985469 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:52.985535 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:52.986351 14634 server_base.cc:1034] running on GCE node
I20260925 01:23:52.987092 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:52.988437 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:52.989677 14634 hybrid_clock.cc:648] HybridClock initialized: now 1790299432989612 us; error 31 us; skew 500 ppm
I20260925 01:23:52.989961 14634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:52 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299432, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:52.992869 14634 init.cc:377] Logged in from keytab as oryx/127.6.236.190@KRBTEST.COM (short username oryx)
I20260925 01:23:52.994216 14634 webserver.cc:460] Webserver started at http://127.6.236.190:33813/ using document root <none> and password file <none>
I20260925 01:23:52.994877 14634 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:52.994961 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:52.995216 14634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:52.998099 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "bd3da72e917643f298dac0b4581bd335"
format_stamp: "Formatted at 2026-09-25 01:23:52 on dist-test-slave-s0b2"
server_key: "66bcecdf2f79a4a849b0a29be775c1b4"
server_key_iv: "ac3f5d18a5bbc9fdc653cd24a575a896"
server_key_version: "encryptionkey@0"
I20260925 01:23:52.998875 14634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "bd3da72e917643f298dac0b4581bd335"
format_stamp: "Formatted at 2026-09-25 01:23:52 on dist-test-slave-s0b2"
server_key: "66bcecdf2f79a4a849b0a29be775c1b4"
server_key_iv: "ac3f5d18a5bbc9fdc653cd24a575a896"
server_key_version: "encryptionkey@0"
I20260925 01:23:53.003870 14634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:23:53.007313 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:53.008611 14634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:23:53.008747 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "bd3da72e917643f298dac0b4581bd335"
format_stamp: "Formatted at 2026-09-25 01:23:52 on dist-test-slave-s0b2"
server_key: "66bcecdf2f79a4a849b0a29be775c1b4"
server_key_iv: "ac3f5d18a5bbc9fdc653cd24a575a896"
server_key_version: "encryptionkey@0"
I20260925 01:23:53.008919 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:53.028232 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:53.032371 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:53.032613 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:53.051237 14634 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:43867
I20260925 01:23:53.051256 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:43867 every 8 connection(s)
I20260925 01:23:53.052522 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:23:53.056121 14711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:53.061815 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14634
I20260925 01:23:53.061995 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:23:53.062668 7090 external_mini_cluster.cc:1428] Setting key 4c96c6f505538e82639a88b1cd5feb9e
I20260925 01:23:53.066231 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Bootstrap starting.
I20260925 01:23:53.069721 14711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:53.070755 14711 log.cc:824] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:53.073580 14711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: No bootstrap required, opened a new log
Sep 25 01:23:53 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:53.082454 14711 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } }
I20260925 01:23:53.082886 14711 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:53.083010 14711 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3da72e917643f298dac0b4581bd335, State: Initialized, Role: FOLLOWER
I20260925 01:23:53.083676 14711 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } }
I20260925 01:23:53.083900 14711 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:23:53.084048 14711 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:23:53.084215 14711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:53.084685 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:53.064573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.6.236.190:43867)
0925 01:23:53.065316 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:53.065327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:53.065367 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:23:53.067049 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:53.067070 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:53.067432 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:53.067758 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:53.069392 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.070747 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.072182 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.072838 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.072979 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:53.077472 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:53.077481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:53.077496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:53.077539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:53.080500 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.081281 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.081287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.081293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.081473 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.081869 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.081872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.081874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.082420 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:53.082648 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:53.083197 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 01:23:53.083576 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":327,"threads_started":1}
I20260925 01:23:53.086817 14711 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } }
I20260925 01:23:53.087384 14711 leader_election.cc:302] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd3da72e917643f298dac0b4581bd335; no voters:
I20260925 01:23:53.087970 14711 leader_election.cc:288] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:23:53.088085 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:53.088743 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [term 1 LEADER]: Becoming Leader. State: Replica: bd3da72e917643f298dac0b4581bd335, State: Running, Role: LEADER
I20260925 01:23:53.089259 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } }
I20260925 01:23:53.089831 14711 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:23:53.092425 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd3da72e917643f298dac0b4581bd335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } } }
I20260925 01:23:53.092852 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:53.092989 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd3da72e917643f298dac0b4581bd335. Latest consensus state: current_term: 1 leader_uuid: "bd3da72e917643f298dac0b4581bd335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3da72e917643f298dac0b4581bd335" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 43867 } } }
I20260925 01:23:53.093863 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335 [sys.catalog]: This master's current role is: LEADER
I20260925 01:23:53.094911 14726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:23:53.101976 14726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:23:53.111811 14726 catalog_manager.cc:1347] Generated new cluster ID: 2659bdad02c4443d9dfe8c2281bb0d0b
I20260925 01:23:53.111938 14726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:23:53.132164 14726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:23:53.133473 14726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:23:53.140399 14726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Generated new TSK 0
I20260925 01:23:53.141196 14726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
I20260925 01:23:53.205384 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:53.535214 14737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:53.535496 14737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:53.535634 14737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:53.542965 14737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:53.543081 14737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:53.543172 14737 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:53.543251 14737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:23:53.552728 14737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:43867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:53.553886 14737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:53.555142 14737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:53.567484 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:53.567458 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:53.567433 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:53.568327 14737 server_base.cc:1034] running on GCE node
I20260925 01:23:53.568899 14737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:53.569836 14737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:53.571038 14737 hybrid_clock.cc:648] HybridClock initialized: now 1790299433570960 us; error 56 us; skew 500 ppm
I20260925 01:23:53.571322 14737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:53 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299433, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:53.574344 14737 init.cc:377] Logged in from keytab as oryx/127.6.236.129@KRBTEST.COM (short username oryx)
I20260925 01:23:53.575707 14737 webserver.cc:460] Webserver started at http://127.6.236.129:41597/ using document root <none> and password file <none>
I20260925 01:23:53.576390 14737 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:53.576494 14737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:53.576818 14737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:53.579694 14737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "f66afdd71e2b4ab3b1781109bd897431"
format_stamp: "Formatted at 2026-09-25 01:23:53 on dist-test-slave-s0b2"
server_key: "0a1c5117c98e09ab4c7cb853f956bb80"
server_key_iv: "a1aeefe3df0e1c9738f17fa75274080b"
server_key_version: "encryptionkey@0"
I20260925 01:23:53.580462 14737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "f66afdd71e2b4ab3b1781109bd897431"
format_stamp: "Formatted at 2026-09-25 01:23:53 on dist-test-slave-s0b2"
server_key: "0a1c5117c98e09ab4c7cb853f956bb80"
server_key_iv: "a1aeefe3df0e1c9738f17fa75274080b"
server_key_version: "encryptionkey@0"
I20260925 01:23:53.585577 14737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:23:53.588986 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:53.590372 14737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:23:53.590518 14737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "f66afdd71e2b4ab3b1781109bd897431"
format_stamp: "Formatted at 2026-09-25 01:23:53 on dist-test-slave-s0b2"
server_key: "0a1c5117c98e09ab4c7cb853f956bb80"
server_key_iv: "a1aeefe3df0e1c9738f17fa75274080b"
server_key_version: "encryptionkey@0"
I20260925 01:23:53.590758 14737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:53.618718 14737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:53.622835 14737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:53.623063 14737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:53.625723 14737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:53.625806 14737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:53.625931 14737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:53.625989 14737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:53 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299433, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:53.649326 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:53.627002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.129:39479)
0925 01:23:53.627839 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:53.627895 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:53.630263 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:53.631412 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:53.631423 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:53.632162 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:53.633019 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:53.633045 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.634632 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.634638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:53.636579 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:53.636597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.637595 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.637606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:53.637848 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:53.638620 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:53.638641 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:53.641699 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:53.644848 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:53.644860 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:53.644879 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:53.645493 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:53.645919 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:53.645924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:53.645928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:53.646129 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:53.646776 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:53.646785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:53.647213 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 01:23:53.647577 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 01:23:53.648110 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":331,"threads_started":1}
I20260925 01:23:53.650230 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:53.626489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:39479 (local address 127.6.236.190:43867)
0925 01:23:53.628652 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:53.628658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:53.628681 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:23:53.630621 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:53.630628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:53.630721 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:53.630807 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:53.633283 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.634459 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.636789 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.637352 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.637468 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:53.641910 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:53.641917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:53.641922 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:53.641967 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:53.644718 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.645631 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.645637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.645641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.645796 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.646224 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.646228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.646231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.646616 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:53.646773 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:53.649829 (+ 3056us) server_negotiation.cc:299] Negotiation successful
0925 01:23:53.650000 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2026,"thread_start_us":1908,"threads_started":1}
I20260925 01:23:53.664987 14737 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:37589
I20260925 01:23:53.665081 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:37589 every 8 connection(s)
I20260925 01:23:53.666172 14737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:23:53.668175 14873 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43867
I20260925 01:23:53.668385 14873 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:53.669019 14873 heartbeater.cc:511] Master 127.6.236.190:43867 requested a full tablet report, sending...
I20260925 01:23:53.669438 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:53.656324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42095 (local address 127.6.236.190:43867)
0925 01:23:53.656481 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:53.656487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:53.657224 (+ 737us) server_negotiation.cc:407] Connection header received
0925 01:23:53.657348 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:53.657354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:53.657437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:53.657541 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:53.658387 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.659292 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.661267 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.661826 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.661942 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:53.663980 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:53.663987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:53.663991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:53.664031 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:53.666676 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.667530 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.667535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.667538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.667694 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:53.668036 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:53.668040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:53.668042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:53.668367 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:53.668533 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:53.668904 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 01:23:53.669106 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:23:53.669549 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:53.656442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.129:42095)
0925 01:23:53.657135 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:53.657150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:53.657253 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:53.657546 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:53.657549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:53.657944 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:53.658227 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:53.658260 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.659461 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.659465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:53.661080 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:53.661092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:53.662176 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:53.662184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:53.662387 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:53.662994 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:53.663017 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:53.663795 (+ 778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:53.666794 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:53.666799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:53.666803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:53.667421 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:53.667799 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:53.667803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:53.667807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:53.667935 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:53.668483 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:53.668488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:53.668669 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 01:23:53.668793 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 01:23:53.669120 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":388,"threads_started":1}
I20260925 01:23:53.671873 14666 ts_manager.cc:194] Registered new tserver with Master: f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589)
I20260925 01:23:53.671958 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14737
I20260925 01:23:53.672128 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:23:53.672919 7090 external_mini_cluster.cc:1428] Setting key 20367b3de3a4238166569279d37c91aa
I20260925 01:23:53.673496 14666 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:39479
WARNING: no policy specified for oryx/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
I20260925 01:23:53.714576 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:54.022166 14877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:54.022460 14877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:54.022593 14877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:54.029517 14877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:54.029629 14877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:54.029747 14877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:54.029853 14877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:23:54.039311 14877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:43867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:54.040488 14877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:54.041777 14877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:54.053817 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:54.054023 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:54.054502 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:54.055368 14877 server_base.cc:1034] running on GCE node
I20260925 01:23:54.056051 14877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:54.057078 14877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:54.058346 14877 hybrid_clock.cc:648] HybridClock initialized: now 1790299434058278 us; error 34 us; skew 500 ppm
I20260925 01:23:54.058701 14877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:54.066557 14877 init.cc:377] Logged in from keytab as oryx/127.6.236.130@KRBTEST.COM (short username oryx)
I20260925 01:23:54.067996 14877 webserver.cc:460] Webserver started at http://127.6.236.130:44173/ using document root <none> and password file <none>
I20260925 01:23:54.068619 14877 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:54.068699 14877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:54.068959 14877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:54.071688 14877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "67220d865fe7492f845a84569ee20624"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "a8f8bf01d123df99fe469c7229326e9b"
server_key_iv: "89b90286a5b5ee299e2b876a16dc8148"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.072381 14877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "67220d865fe7492f845a84569ee20624"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "a8f8bf01d123df99fe469c7229326e9b"
server_key_iv: "89b90286a5b5ee299e2b876a16dc8148"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.076982 14877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 01:23:54.080188 14892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:54.081436 14877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:54.081588 14877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "67220d865fe7492f845a84569ee20624"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "a8f8bf01d123df99fe469c7229326e9b"
server_key_iv: "89b90286a5b5ee299e2b876a16dc8148"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.081789 14877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:54.102865 14877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:54.106451 14877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:54.106693 14877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:54.109249 14877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:54.109342 14877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:54.109483 14877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:54.109596 14877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:54.131062 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.109634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:52997 (local address 127.6.236.190:43867)
0925 01:23:54.109802 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:54.109807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:54.111184 (+ 1377us) server_negotiation.cc:407] Connection header received
0925 01:23:54.113263 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:54.113268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:54.113379 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:54.113463 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:54.115391 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.116377 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.119122 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.119654 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.119781 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.124406 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:54.124411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:54.124414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:54.124447 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:54.127558 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.128556 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.128560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.128562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.128703 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.129174 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.129177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.129179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.129458 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:54.129564 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:54.130605 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0925 01:23:54.130816 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:54.132748 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.110326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.130:52997)
0925 01:23:54.111035 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:54.111070 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:54.112994 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:54.113694 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:54.113702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:54.114392 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:54.115211 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.115245 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.116529 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.116538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.118951 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.118969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.120100 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.120117 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.120395 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.121106 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:54.121134 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:54.124185 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:54.127740 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.127750 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.127772 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.128422 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.128799 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.128803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.128807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.129076 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.129561 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:54.129570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:54.130061 (+ 491us) client_negotiation.cc:769] Sending connection context
0925 01:23:54.130412 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 01:23:54.131001 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260925 01:23:54.150870 14877 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:37413
I20260925 01:23:54.150888 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:37413 every 8 connection(s)
I20260925 01:23:54.152207 14877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:23:54.153466 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.138645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:50251 (local address 127.6.236.190:43867)
0925 01:23:54.138830 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:54.138836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:54.140103 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 01:23:54.140368 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:54.140374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:54.140463 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:54.140572 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:54.142275 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.143039 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.144895 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.145435 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.145555 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.148002 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:54.148009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:54.148022 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:54.148075 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:54.150880 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.151515 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.151520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.151523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.151656 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.152002 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.152008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.152011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.152379 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:54.152530 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:54.152920 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 01:23:54.153137 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:23:54.153647 14976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.138701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.130:50251)
0925 01:23:54.139999 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:54.140018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:54.140169 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:54.140794 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:54.140799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:54.141521 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:54.142026 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.142044 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.143175 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.143180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.144757 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.144776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.145812 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.145819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.145947 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.146718 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:54.146736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:54.147830 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:54.151013 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.151017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.151022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.151402 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.151762 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.151767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.151771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.151896 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.152505 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:54.152510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:54.152689 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 01:23:54.152813 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 01:23:54.153173 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":484,"threads_started":1}
I20260925 01:23:54.155179 15012 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43867
I20260925 01:23:54.155400 15012 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:54.156031 15012 heartbeater.cc:511] Master 127.6.236.190:43867 requested a full tablet report, sending...
I20260925 01:23:54.157301 14665 ts_manager.cc:194] Registered new tserver with Master: 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:54.158097 14665 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:52997
I20260925 01:23:54.158529 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 14877
I20260925 01:23:54.158716 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:23:54.159246 7090 external_mini_cluster.cc:1428] Setting key 82d2952bfb09f5b3d46cb658031844b1
WARNING: no policy specified for oryx/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
I20260925 01:23:54.200655 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:43867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:54.507586 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:54.507854 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:54.507961 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:54.514950 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:54.515058 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:54.515144 15016 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:23:54.515223 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:23:54.524431 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.236.190:43867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:54.525550 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:54.526814 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:54.539037 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:54.539104 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:54.539124 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:54.539812 15016 server_base.cc:1034] running on GCE node
I20260925 01:23:54.540371 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:54.541209 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:54.542399 15016 hybrid_clock.cc:648] HybridClock initialized: now 1790299434542319 us; error 61 us; skew 500 ppm
I20260925 01:23:54.542680 15016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:23:54.545506 15016 init.cc:377] Logged in from keytab as oryx/127.6.236.131@KRBTEST.COM (short username oryx)
I20260925 01:23:54.546865 15016 webserver.cc:460] Webserver started at http://127.6.236.131:41315/ using document root <none> and password file <none>
I20260925 01:23:54.547472 15016 fs_manager.cc:359] Metadata directory not provided
I20260925 01:23:54.547564 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:23:54.547860 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:23:54.550601 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "dc60764d45484c07987b04de1c08901f"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "7c925eaac6fe00c198690631a97ef97a"
server_key_iv: "46e65dd5df05fa1aae8d731f91ed7d1f"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.551337 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "dc60764d45484c07987b04de1c08901f"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "7c925eaac6fe00c198690631a97ef97a"
server_key_iv: "46e65dd5df05fa1aae8d731f91ed7d1f"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.556073 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 01:23:54.559197 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:54.560415 15016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:23:54.560583 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "dc60764d45484c07987b04de1c08901f"
format_stamp: "Formatted at 2026-09-25 01:23:54 on dist-test-slave-s0b2"
server_key: "7c925eaac6fe00c198690631a97ef97a"
server_key_iv: "46e65dd5df05fa1aae8d731f91ed7d1f"
server_key_version: "encryptionkey@0"
I20260925 01:23:54.560777 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:23:54.579341 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:23:54.583428 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:23:54.583668 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:23:54.586162 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:23:54.586264 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:23:54.586419 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:23:54.586520 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:54.607105 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.586599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57079 (local address 127.6.236.190:43867)
0925 01:23:54.586778 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:54.586784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:54.588098 (+ 1314us) server_negotiation.cc:407] Connection header received
0925 01:23:54.590211 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:54.590217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:54.590344 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:54.590431 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:54.592379 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.593588 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.596240 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.596661 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.596771 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.600960 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:54.600964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:54.600968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:54.600999 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:54.604155 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.604822 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.604828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.604832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.604962 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.605368 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.605374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.605377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.605646 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:54.605782 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:54.606504 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 01:23:54.606744 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:23:54.608060 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.587112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.131:57079)
0925 01:23:54.587921 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:54.587978 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:54.589930 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:54.590679 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:54.590690 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:54.591354 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:54.592176 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.592206 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.593749 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.593753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.596101 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.596113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.596876 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.596883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.597215 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.597851 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:54.597871 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:54.600677 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:54.604284 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.604293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.604312 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.604713 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.605068 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.605073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.605077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.605266 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.605770 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:54.605777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:54.606093 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 01:23:54.606363 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 01:23:54.606867 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":318,"threads_started":1}
I20260925 01:23:54.627202 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.613610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:49757 (local address 127.6.236.190:43867)
0925 01:23:54.613775 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:54.613781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:54.614435 (+ 654us) server_negotiation.cc:407] Connection header received
0925 01:23:54.614657 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:54.614662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:54.614749 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:54.614849 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:54.615944 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.616594 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.618495 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.618991 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.619113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.621585 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:54.621591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:54.621595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:54.621628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:54.624378 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.625212 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.625218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.625222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.625374 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:54.625834 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:54.625840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:54.625843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:54.626207 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:54.626366 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:54.626794 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 01:23:54.626952 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:23:54.627547 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:54.613641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.6.236.131:49757)
0925 01:23:54.614332 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:54.614353 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:54.614536 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:54.614897 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:54.614905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:54.615456 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:54.615819 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.615831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.616709 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.616714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.618353 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:54.618367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.619316 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.619323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:54.619553 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:54.620178 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:54.620204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:54.621388 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:54.624495 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.624501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.624504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.625078 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.625518 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:54.625527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:54.625530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:54.625708 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:54.626342 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:54.626347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:54.626564 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 01:23:54.626723 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 01:23:54.627070 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":318,"threads_started":1}
I20260925 01:23:54.638041 15016 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44835
I20260925 01:23:54.638077 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44835 every 8 connection(s)
I20260925 01:23:54.639164 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:23:54.641191 15151 heartbeater.cc:347] Connected to a master server at 127.6.236.190:43867
I20260925 01:23:54.641391 15151 heartbeater.cc:464] Registering TS with master...
I20260925 01:23:54.641996 15151 heartbeater.cc:511] Master 127.6.236.190:43867 requested a full tablet report, sending...
I20260925 01:23:54.643262 14665 ts_manager.cc:194] Registered new tserver with Master: dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:54.644011 14665 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:57079
I20260925 01:23:54.647974 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15016
I20260925 01:23:54.648128 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:23:54.648569 7090 external_mini_cluster.cc:1428] Setting key 56b87480ecd42aebb2432c1b8354d350
I20260925 01:23:54.651170 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:23:54.676508 14873 heartbeater.cc:503] Master 127.6.236.190:43867 was elected leader, sending a full tablet report...
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:23:54 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:23:55.160851 15012 heartbeater.cc:503] Master 127.6.236.190:43867 was elected leader, sending a full tablet report...
I20260925 01:23:55.646171 15151 heartbeater.cc:503] Master 127.6.236.190:43867 was elected leader, sending a full tablet report...
W20260925 01:23:55.657738 15159 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.964s user 0.002s sys 0.011s
W20260925 01:23:55.658763 14761 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:54.688260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.6.236.190:43867)
0925 01:23:54.688479 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:54.688485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:54.688720 (+ 235us) server_negotiation.cc:407] Connection header received
0925 01:23:54.688901 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:54.688905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:54.688967 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:54.689055 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:54.690378 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.691173 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.692347 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:54.692741 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:54.692823 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.658529 (+965706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60752: BlockingRecv error: recv got EOF from 127.0.0.1:60752 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 01:23:55 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299434, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:55.678489 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.665648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.6.236.190:43867)
0925 01:23:55.665838 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.665844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.666121 (+ 277us) server_negotiation.cc:407] Connection header received
0925 01:23:55.666289 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.666294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.666390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.666510 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:55.667739 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.668475 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.669726 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.670152 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.670267 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.673799 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:55.673803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:55.673807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:55.673838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:55.676330 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:55.676934 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:55.676937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:55.676940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:55.677068 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:55.677389 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:55.677392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:55.677394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:55.677630 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:55.677856 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.678039 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.678208 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:23:55.682428 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:55.689204 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:55.711953 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.700737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44835 (local address 127.0.0.1:54316)
0925 01:23:55.702040 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:55.702082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:55.702296 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:55.703488 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:55.703497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:55.703530 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:55.703998 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.704010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.706076 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.706082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.708608 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.708633 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.710936 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.710943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.711180 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.711233 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 01:23:55.711359 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 01:23:55.711564 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"threads_started":1}
I20260925 01:23:55.712853 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.700947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.6.236.131:44835)
0925 01:23:55.701523 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.701532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.702521 (+ 989us) server_negotiation.cc:407] Connection header received
0925 01:23:55.702649 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.702657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.702920 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.703100 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:55.704299 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.705879 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.708808 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.710428 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.710567 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.710782 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.712359 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.712551 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":251,"threads_started":1}
I20260925 01:23:55.719023 15082 tablet_service.cc:1467] Processing CreateTablet for tablet dd55604f0d494ad38b19a71673a5ca13 (DEFAULT_TABLE table=test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:55.720592 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd55604f0d494ad38b19a71673a5ca13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:55.725939 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.700862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37413 (local address 127.0.0.1:41368)
0925 01:23:55.713232 (+ 12370us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:55.713252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:55.713352 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:55.718772 (+ 5420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:55.718776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:55.718800 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:55.719031 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.719052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.720945 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.720950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.723190 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.723221 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.725125 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.725132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.725322 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.725357 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:23:55.725423 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 01:23:55.725552 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12267,"mutex_wait_us":23,"thread_start_us":2105,"threads_started":1}
I20260925 01:23:55.726639 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.701508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.0.0.1:60126)
0925 01:23:55.714714 (+ 13206us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:55.714730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:55.714828 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:55.718118 (+ 3290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:55.718123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:55.718162 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:55.718577 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.718588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.720546 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.720552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.723494 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.723504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.725689 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.725696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.725894 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.725932 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:23:55.726007 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 01:23:55.726294 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13087,"mutex_wait_us":1,"threads_started":1}
I20260925 01:23:55.727553 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.6.236.129:37589)
0925 01:23:55.716513 (+ 14126us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.716521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.716550 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:23:55.716643 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.716650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.716889 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.717082 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:55.718783 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.720368 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.723655 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.725176 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.725314 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.725528 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.727061 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.727262 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13971,"thread_start_us":13832,"threads_started":1}
I20260925 01:23:55.727939 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.701842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.6.236.130:37413)
0925 01:23:55.717582 (+ 15740us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.717590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.717615 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:23:55.717709 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.717717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.717913 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.719208 (+ 1295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:55.719389 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.720771 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.723534 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.724929 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.726183 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.726394 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.727510 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.727670 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15594,"thread_start_us":15466,"threads_started":1}
I20260925 01:23:55.731765 14942 tablet_service.cc:1467] Processing CreateTablet for tablet dd55604f0d494ad38b19a71673a5ca13 (DEFAULT_TABLE table=test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:55.731765 14803 tablet_service.cc:1467] Processing CreateTablet for tablet dd55604f0d494ad38b19a71673a5ca13 (DEFAULT_TABLE table=test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:55.733171 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd55604f0d494ad38b19a71673a5ca13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:55.733174 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd55604f0d494ad38b19a71673a5ca13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:55.733884 15176 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Bootstrap starting.
I20260925 01:23:55.737048 15176 tablet_bootstrap.cc:654] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:55.738435 15176 log.cc:824] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:55.740772 15176 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: No bootstrap required, opened a new log
I20260925 01:23:55.741001 15176 ts_tablet_manager.cc:1397] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 01:23:55.747121 15178 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Bootstrap starting.
I20260925 01:23:55.747824 15179 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Bootstrap starting.
I20260925 01:23:55.748456 15176 raft_consensus.cc:348] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.748878 15176 raft_consensus.cc:374] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:55.748983 15176 raft_consensus.cc:729] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60764d45484c07987b04de1c08901f, State: Initialized, Role: FOLLOWER
I20260925 01:23:55.749594 15176 consensus_queue.cc:260] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.751211 15176 ts_tablet_manager.cc:1428] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260925 01:23:55.751500 15178 tablet_bootstrap.cc:654] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:55.752856 15178 log.cc:824] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:55.754565 15179 tablet_bootstrap.cc:654] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:55.755721 15179 log.cc:824] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Log is configured to *not* fsync() on all Append() calls
I20260925 01:23:55.756549 15178 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: No bootstrap required, opened a new log
I20260925 01:23:55.756789 15178 ts_tablet_manager.cc:1397] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260925 01:23:55.758060 15179 tablet_bootstrap.cc:492] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: No bootstrap required, opened a new log
I20260925 01:23:55.758316 15179 ts_tablet_manager.cc:1397] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:23:55.763446 15178 raft_consensus.cc:348] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.763859 15178 raft_consensus.cc:374] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:55.763962 15178 raft_consensus.cc:729] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Initialized, Role: FOLLOWER
I20260925 01:23:55.764599 15178 consensus_queue.cc:260] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.765772 15179 raft_consensus.cc:348] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.766182 15178 ts_tablet_manager.cc:1428] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Time spent starting tablet: real 0.009s user 0.008s sys 0.003s
I20260925 01:23:55.766196 15179 raft_consensus.cc:374] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:55.766309 15179 raft_consensus.cc:729] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67220d865fe7492f845a84569ee20624, State: Initialized, Role: FOLLOWER
I20260925 01:23:55.766772 15179 consensus_queue.cc:260] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.768105 15179 ts_tablet_manager.cc:1428] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 01:23:55.888067 15183 raft_consensus.cc:482] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:55.888422 15183 raft_consensus.cc:504] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.890210 15183 leader_election.cc:288] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc60764d45484c07987b04de1c08901f (127.6.236.131:44835), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
W20260925 01:23:55.891462 15152 tablet.cc:2369] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:55.897951 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.891327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:35143 (local address 127.6.236.131:44835)
0925 01:23:55.891505 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.891509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.892602 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 01:23:55.892780 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.892785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.892871 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.892937 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:55.893584 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.894301 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.896288 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.897069 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.897199 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.897271 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.897614 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.897736 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:55.897984 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.890741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:60293 (local address 127.6.236.130:37413)
0925 01:23:55.890937 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.890943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.892724 (+ 1781us) server_negotiation.cc:407] Connection header received
0925 01:23:55.892872 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.892875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.892954 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.893042 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:55.893689 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.894497 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.896379 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.897100 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.897247 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.897317 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.897629 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.897756 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:23:55.898144 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.890498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44835 (local address 127.6.236.129:35143)
0925 01:23:55.892481 (+ 1983us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:55.892504 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:55.892644 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:55.892967 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:55.892971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:55.892995 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:55.893437 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.893452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.894420 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.894425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.896140 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.896158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.897298 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.897304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.897458 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.897498 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:23:55.897567 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 01:23:55.897691 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1834,"mutex_wait_us":59,"thread_start_us":602,"threads_started":1}
I20260925 01:23:55.898175 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.890965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37413 (local address 127.6.236.129:60293)
0925 01:23:55.892613 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:55.892630 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:55.892760 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:55.893079 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:55.893083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:55.893115 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:55.893488 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.893498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.894641 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.894647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.896232 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:55.896246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.897342 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.897348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:55.897461 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.897509 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 01:23:55.897592 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 01:23:55.897720 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":17,"spinlock_wait_cycles":5504,"thread_start_us":1377,"threads_started":1}
I20260925 01:23:55.899089 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd55604f0d494ad38b19a71673a5ca13" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f" is_pre_election: true
I20260925 01:23:55.899096 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd55604f0d494ad38b19a71673a5ca13" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624" is_pre_election: true
I20260925 01:23:55.899523 15106 raft_consensus.cc:2454] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 0.
I20260925 01:23:55.899521 14962 raft_consensus.cc:2454] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 0.
I20260925 01:23:55.900066 14753 leader_election.cc:302] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:55.900411 15183 raft_consensus.cc:2781] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:55.900534 15183 raft_consensus.cc:482] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:55.900636 15183 raft_consensus.cc:3037] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:55.902981 15183 raft_consensus.cc:504] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.903599 15183 leader_election.cc:288] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Requested vote from peers dc60764d45484c07987b04de1c08901f (127.6.236.131:44835), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:55.903982 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd55604f0d494ad38b19a71673a5ca13" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f"
I20260925 01:23:55.904040 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd55604f0d494ad38b19a71673a5ca13" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624"
I20260925 01:23:55.904143 15106 raft_consensus.cc:3037] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:55.904232 14962 raft_consensus.cc:3037] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Advancing to term 1
W20260925 01:23:55.906140 15013 tablet.cc:2369] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:55.906315 15106 raft_consensus.cc:2454] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 1.
I20260925 01:23:55.906327 14962 raft_consensus.cc:2454] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 1.
I20260925 01:23:55.906664 14753 leader_election.cc:302] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:55.906946 15183 raft_consensus.cc:2781] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:55.907366 15183 raft_consensus.cc:686] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 1 LEADER]: Becoming Leader. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Running, Role: LEADER
I20260925 01:23:55.907789 15183 consensus_queue.cc:237] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } }
I20260925 01:23:55.911824 14666 catalog_manager.cc:5495] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 reported cstate change: term changed from 0 to 1, leader changed from <none> to f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } health_report { overall_health: HEALTHY } } }
W20260925 01:23:55.919093 14874 tablet.cc:2369] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:23:55.960570 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:55.955521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.6.236.129:37589)
0925 01:23:55.955723 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:55.955729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:55.955747 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:23:55.955932 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:55.955936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:55.956026 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:55.956162 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:55.956502 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.957197 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.958558 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:55.958955 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:55.959062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:55.959474 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:55.959978 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:55.960117 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:55.960226 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 01:23:55.960304 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:23:55.968433 15106 raft_consensus.cc:1261] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:55.968439 14962 raft_consensus.cc:1261] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:55.969311 15187 consensus_queue.cc:1035] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:55.969638 15183 consensus_queue.cc:1035] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:55.979502 15190 mvcc.cc:205] Tried to move back new op lower bound from 7333066489717129216 to 7333066489481490432. Current Snapshot: MvccSnapshot[applied={T|T < 7333066489717129216}]
I20260925 01:23:55.992775 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:55.992992 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:55.996447 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Sending DeleteTablet for 3 replicas of tablet dd55604f0d494ad38b19a71673a5ca13
I20260925 01:23:55.997551 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet dd55604f0d494ad38b19a71673a5ca13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:55 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54316
I20260925 01:23:55.997560 14942 tablet_service.cc:1514] Processing DeleteTablet for tablet dd55604f0d494ad38b19a71673a5ca13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:55 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260925 01:23:55.997591 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet dd55604f0d494ad38b19a71673a5ca13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:55 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60126
I20260925 01:23:56.001317 14665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:56.002768 14665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:56.009985 15201 tablet_replica.cc:330] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: stopping tablet replica
I20260925 01:23:56.010694 15201 raft_consensus.cc:2229] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:56.011188 14941 tablet_service.cc:1467] Processing CreateTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b (DEFAULT_TABLE table=test-table [id=41ef4ba927f54b5988332ee3910273aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.011336 15201 raft_consensus.cc:2258] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.011816 14941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d468d14c1f834b9e9054b2f73c1a0d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.013665 15201 ts_tablet_manager.cc:1905] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.015338 14802 tablet_service.cc:1467] Processing CreateTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b (DEFAULT_TABLE table=test-table [id=41ef4ba927f54b5988332ee3910273aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.015925 14802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d468d14c1f834b9e9054b2f73c1a0d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.019346 15082 tablet_service.cc:1467] Processing CreateTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b (DEFAULT_TABLE table=test-table [id=41ef4ba927f54b5988332ee3910273aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.021378 15200 tablet_replica.cc:330] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: stopping tablet replica
I20260925 01:23:56.021746 15200 raft_consensus.cc:2229] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:56.022167 15200 raft_consensus.cc:2258] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.020866 15201 ts_tablet_manager.cc:1918] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:56.022578 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d468d14c1f834b9e9054b2f73c1a0d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.022583 15201 log.cc:1196] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/dd55604f0d494ad38b19a71673a5ca13
I20260925 01:23:56.023263 15201 ts_tablet_manager.cc:1939] T dd55604f0d494ad38b19a71673a5ca13 P dc60764d45484c07987b04de1c08901f: Deleting consensus metadata
I20260925 01:23:56.024318 15200 ts_tablet_manager.cc:1905] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.025159 14650 catalog_manager.cc:4841] TS dc60764d45484c07987b04de1c08901f (127.6.236.131:44835): tablet dd55604f0d494ad38b19a71673a5ca13 (table test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]) successfully deleted
I20260925 01:23:56.028105 15178 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Bootstrap starting.
I20260925 01:23:56.030017 15178 tablet_bootstrap.cc:654] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.030728 15179 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Bootstrap starting.
I20260925 01:23:56.032379 15179 tablet_bootstrap.cc:654] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.033948 15202 tablet_replica.cc:330] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: stopping tablet replica
I20260925 01:23:56.034384 15200 ts_tablet_manager.cc:1918] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:56.034498 15202 raft_consensus.cc:2229] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:56.035238 15202 raft_consensus.cc:2258] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.035691 15200 log.cc:1196] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/dd55604f0d494ad38b19a71673a5ca13
I20260925 01:23:56.036278 15200 ts_tablet_manager.cc:1939] T dd55604f0d494ad38b19a71673a5ca13 P 67220d865fe7492f845a84569ee20624: Deleting consensus metadata
I20260925 01:23:56.037278 15202 ts_tablet_manager.cc:1905] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.038975 14653 catalog_manager.cc:4841] TS 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): tablet dd55604f0d494ad38b19a71673a5ca13 (table test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]) successfully deleted
I20260925 01:23:56.041184 15176 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Bootstrap starting.
I20260925 01:23:56.042800 15176 tablet_bootstrap.cc:654] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.044647 15176 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: No bootstrap required, opened a new log
I20260925 01:23:56.044775 15176 ts_tablet_manager.cc:1397] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:23:56.045629 15176 raft_consensus.cc:348] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.045799 15176 raft_consensus.cc:374] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.046139 15176 raft_consensus.cc:729] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60764d45484c07987b04de1c08901f, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.046510 15176 consensus_queue.cc:260] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.047106 15176 ts_tablet_manager.cc:1428] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:56.047305 15179 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: No bootstrap required, opened a new log
I20260925 01:23:56.047398 15202 ts_tablet_manager.cc:1918] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:56.047499 15179 ts_tablet_manager.cc:1397] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260925 01:23:56.047616 15202 log.cc:1196] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/dd55604f0d494ad38b19a71673a5ca13
I20260925 01:23:56.048183 15202 ts_tablet_manager.cc:1939] T dd55604f0d494ad38b19a71673a5ca13 P f66afdd71e2b4ab3b1781109bd897431: Deleting consensus metadata
I20260925 01:23:56.048389 15179 raft_consensus.cc:348] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.048671 15179 raft_consensus.cc:374] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.048810 15179 raft_consensus.cc:729] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67220d865fe7492f845a84569ee20624, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.049086 15179 consensus_queue.cc:260] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.050155 14651 catalog_manager.cc:4841] TS f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589): tablet dd55604f0d494ad38b19a71673a5ca13 (table test-table [id=8e78ee1aff5c494e9c9ddc743f1875c3]) successfully deleted
I20260925 01:23:56.050729 15179 ts_tablet_manager.cc:1428] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 01:23:56.052376 15178 tablet_bootstrap.cc:492] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: No bootstrap required, opened a new log
I20260925 01:23:56.052575 15178 ts_tablet_manager.cc:1397] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.008s
I20260925 01:23:56.053449 15178 raft_consensus.cc:348] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.053740 15178 raft_consensus.cc:374] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.053880 15178 raft_consensus.cc:729] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.054227 15178 consensus_queue.cc:260] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.054908 15178 ts_tablet_manager.cc:1428] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:56.114941 15184 raft_consensus.cc:482] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:56.115134 15184 raft_consensus.cc:504] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.116616 15184 leader_election.cc:288] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:56.124155 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.117112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:34287 (local address 127.6.236.129:37589)
0925 01:23:56.117286 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.117292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.118123 (+ 831us) server_negotiation.cc:407] Connection header received
0925 01:23:56.118200 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.118206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.118331 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.118416 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:56.119186 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.120317 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.122179 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.123133 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.123282 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.123356 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.123687 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.123822 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:23:56.124174 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.116881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.6.236.130:34287)
0925 01:23:56.117976 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.117995 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.118105 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.118553 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.118558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.118589 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:56.119005 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.119015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.120498 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.120503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.122029 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.122042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.123331 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.123338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.123512 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.123548 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.123645 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.123781 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":458,"spinlock_wait_cycles":896,"thread_start_us":392,"threads_started":1}
I20260925 01:23:56.125136 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.117079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:40677 (local address 127.6.236.131:44835)
0925 01:23:56.117266 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.117273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.117857 (+ 584us) server_negotiation.cc:407] Connection header received
0925 01:23:56.118145 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.118151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.118232 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.118346 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:56.119227 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.120317 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.122911 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.123989 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.124143 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.124210 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.124746 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.124873 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:56.125172 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.116874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44835 (local address 127.6.236.130:40677)
0925 01:23:56.117744 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.117763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.117974 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.118517 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.118522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.118551 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:56.118996 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.119010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.120460 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.120465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.122735 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.122750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.124367 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.124372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.124546 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.124579 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.124700 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.124816 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":22,"spinlock_wait_cycles":8320,"thread_start_us":550,"threads_started":1}
I20260925 01:23:56.125277 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d468d14c1f834b9e9054b2f73c1a0d2b" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431" is_pre_election: true
I20260925 01:23:56.125736 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d468d14c1f834b9e9054b2f73c1a0d2b" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f" is_pre_election: true
I20260925 01:23:56.125736 14823 raft_consensus.cc:2454] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 0.
I20260925 01:23:56.125952 15106 raft_consensus.cc:2454] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 0.
I20260925 01:23:56.126416 14893 leader_election.cc:302] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:56.126820 15184 raft_consensus.cc:2781] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:56.126930 15184 raft_consensus.cc:482] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:56.127022 15184 raft_consensus.cc:3037] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.129055 15184 raft_consensus.cc:504] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.129647 15184 leader_election.cc:288] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 election: Requested vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:56.130014 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d468d14c1f834b9e9054b2f73c1a0d2b" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431"
I20260925 01:23:56.130227 14823 raft_consensus.cc:3037] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.130175 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d468d14c1f834b9e9054b2f73c1a0d2b" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f"
I20260925 01:23:56.130380 15106 raft_consensus.cc:3037] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.133366 14823 raft_consensus.cc:2454] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 1.
I20260925 01:23:56.133483 15106 raft_consensus.cc:2454] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 1.
I20260925 01:23:56.133764 14894 leader_election.cc:302] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:56.134052 15184 raft_consensus.cc:2781] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:56.134510 15184 raft_consensus.cc:686] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 1 LEADER]: Becoming Leader. State: Replica: 67220d865fe7492f845a84569ee20624, State: Running, Role: LEADER
I20260925 01:23:56.134927 15184 consensus_queue.cc:237] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.137918 14665 catalog_manager.cc:5495] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67220d865fe7492f845a84569ee20624 (127.6.236.130). New cstate: current_term: 1 leader_uuid: "67220d865fe7492f845a84569ee20624" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:23:56 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299432, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.190@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:56.202262 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.188720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:43867 (local address 127.0.0.1:60776)
0925 01:23:56.189374 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.189393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.189548 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.189938 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.189943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.190726 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:23:56.190990 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.191001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.192151 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.192156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.193075 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.193087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.193825 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.193832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.193988 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.194685 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:23:56.194701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:23:56.197017 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:23:56.199864 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:56.199876 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:56.199884 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:56.200283 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:56.200585 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:23:56.200590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:23:56.200594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:23:56.200726 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:23:56.201206 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:23:56.201214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:23:56.201389 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.201537 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.201797 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":311,"threads_started":1}
I20260925 01:23:56.202319 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.188771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.6.236.190:43867)
0925 01:23:56.189682 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.189687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.189706 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:56.189746 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.189749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.189813 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.189899 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:56.191159 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.192026 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.193198 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.193635 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.193739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.197212 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:56.197217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:56.197221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:56.197248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:56.199754 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:56.200382 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:56.200388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:56.200391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:56.200496 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:56.200817 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:56.200822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:56.200826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:56.201079 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:56.201224 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.201636 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.201920 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":792,"thread_start_us":654,"threads_started":1}
I20260925 01:23:56.207369 14665 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60776:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:23:56.208338 14665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:56.214874 14803 tablet_service.cc:1467] Processing CreateTablet for tablet cdd23eaca43f47c1a19c7b5dac577111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcf60f4fa61424aac7fe379a227b73d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:56.215483 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd23eaca43f47c1a19c7b5dac577111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.215307 15082 tablet_service.cc:1467] Processing CreateTablet for tablet cdd23eaca43f47c1a19c7b5dac577111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcf60f4fa61424aac7fe379a227b73d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:56.215368 14941 tablet_service.cc:1467] Processing CreateTablet for tablet cdd23eaca43f47c1a19c7b5dac577111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fdcf60f4fa61424aac7fe379a227b73d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:23:56.215808 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd23eaca43f47c1a19c7b5dac577111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.215898 14941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd23eaca43f47c1a19c7b5dac577111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.222553 15178 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: Bootstrap starting.
I20260925 01:23:56.222576 15176 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: Bootstrap starting.
I20260925 01:23:56.223769 15176 tablet_bootstrap.cc:654] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.224267 15178 tablet_bootstrap.cc:654] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.225660 15176 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: No bootstrap required, opened a new log
I20260925 01:23:56.225800 15176 ts_tablet_manager.cc:1397] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:56.226526 15176 raft_consensus.cc:348] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.226634 15179 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: Bootstrap starting.
I20260925 01:23:56.226699 15176 raft_consensus.cc:374] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.226744 15176 raft_consensus.cc:729] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60764d45484c07987b04de1c08901f, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.226931 15176 consensus_queue.cc:260] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.227427 15176 ts_tablet_manager.cc:1428] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:23:56.227312 15180 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.227670 15180 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.228259 15179 tablet_bootstrap.cc:654] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.231006 15179 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: No bootstrap required, opened a new log
I20260925 01:23:56.231171 15179 ts_tablet_manager.cc:1397] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260925 01:23:56.231452 15178 tablet_bootstrap.cc:492] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: No bootstrap required, opened a new log
I20260925 01:23:56.231667 15178 ts_tablet_manager.cc:1397] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20260925 01:23:56.231863 15179 raft_consensus.cc:348] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.232097 15179 raft_consensus.cc:374] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.232178 15179 raft_consensus.cc:729] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67220d865fe7492f845a84569ee20624, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.232406 15179 consensus_queue.cc:260] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.232443 15178 raft_consensus.cc:348] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.232659 15178 raft_consensus.cc:374] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.232746 15178 raft_consensus.cc:729] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.232988 15178 consensus_queue.cc:260] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.233127 15184 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.233376 15184 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.233587 15178 ts_tablet_manager.cc:1428] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:23:56.234093 15187 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.234493 15187 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.234992 15179 ts_tablet_manager.cc:1428] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 01:23:56.479952 15187 raft_consensus.cc:482] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:56.480191 15187 raft_consensus.cc:504] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.480849 15187 leader_election.cc:288] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:56.481389 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd23eaca43f47c1a19c7b5dac577111" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624" is_pre_election: true
I20260925 01:23:56.481411 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd23eaca43f47c1a19c7b5dac577111" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f" is_pre_election: true
I20260925 01:23:56.481628 14962 raft_consensus.cc:2454] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 0.
I20260925 01:23:56.481688 15106 raft_consensus.cc:2454] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 0.
I20260925 01:23:56.482048 14756 leader_election.cc:302] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:56.482334 15187 raft_consensus.cc:2781] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:56.482434 15187 raft_consensus.cc:482] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:56.482533 15187 raft_consensus.cc:3037] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.484571 15187 raft_consensus.cc:504] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.485220 15187 leader_election.cc:288] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Requested vote from peers 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:56.485615 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd23eaca43f47c1a19c7b5dac577111" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624"
I20260925 01:23:56.485818 14962 raft_consensus.cc:3037] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.485743 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd23eaca43f47c1a19c7b5dac577111" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f"
I20260925 01:23:56.485960 15106 raft_consensus.cc:3037] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:56.488626 15106 raft_consensus.cc:2454] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 1.
I20260925 01:23:56.488638 14962 raft_consensus.cc:2454] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 1.
I20260925 01:23:56.488989 14756 leader_election.cc:302] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:56.489229 15187 raft_consensus.cc:2781] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:56.489542 15187 raft_consensus.cc:686] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [term 1 LEADER]: Becoming Leader. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Running, Role: LEADER
I20260925 01:23:56.489799 15187 consensus_queue.cc:237] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.491227 15220 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: TxnStatusTablet state changed. Reason: New leader f66afdd71e2b4ab3b1781109bd897431. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.491433 15220 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:56.491983 15222 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:23:56.492182 15222 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:23:56.492576 14666 catalog_manager.cc:5495] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 reported cstate change: term changed from 0 to 1, leader changed from <none> to f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:56.530802 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.525923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.6.236.129:37589)
0925 01:23:56.526150 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.526156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.526175 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:23:56.526469 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.526473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.526550 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.526630 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:56.526998 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.527753 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.529072 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.529505 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.529613 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.530034 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.530141 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.530215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.530407 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.530482 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 01:23:56.530810 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.525822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.0.0.1:60158)
0925 01:23:56.526088 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.526110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.526288 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.526674 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.526679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.526692 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:56.526868 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.526876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.527874 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.527880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.528915 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.528935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.529653 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.529658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.529809 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.529864 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.530264 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.530294 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.530365 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.530522 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:23:56.534511 14962 raft_consensus.cc:1261] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:56.534564 15106 raft_consensus.cc:1261] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:56.535115 15187 consensus_queue.cc:1035] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:56.535416 15220 consensus_queue.cc:1035] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:56.539693 15180 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: TxnStatusTablet state changed. Reason: New leader f66afdd71e2b4ab3b1781109bd897431. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.539969 15180 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.540918 15184 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: TxnStatusTablet state changed. Reason: New leader f66afdd71e2b4ab3b1781109bd897431. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.541157 15184 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.542659 15226 mvcc.cc:205] Tried to move back new op lower bound from 7333066492039270400 to 7333066491864305664. Current Snapshot: MvccSnapshot[applied={T|T < 7333066492039270400}]
I20260925 01:23:56.544226 15225 mvcc.cc:205] Tried to move back new op lower bound from 7333066492039270400 to 7333066491864305664. Current Snapshot: MvccSnapshot[applied={T|T < 7333066492039270400}]
I20260925 01:23:56.544677 15184 consensus_queue.cc:1035] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:56.546653 15208 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.546994 15208 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P 67220d865fe7492f845a84569ee20624: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.556079 15220 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.556324 15220 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:56.556794 15180 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.557029 15180 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P dc60764d45484c07987b04de1c08901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:23:56.557339 15220 tablet_replica.cc:439] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } }
I20260925 01:23:56.557600 15220 tablet_replica.cc:442] T cdd23eaca43f47c1a19c7b5dac577111 P f66afdd71e2b4ab3b1781109bd897431: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:23:56.562598 15184 consensus_queue.cc:1035] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:56.576468 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.564788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.6.236.130:37413)
0925 01:23:56.571391 (+ 6603us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.571395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.571410 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:23:56.571487 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.571491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.571581 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.571727 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:56.572217 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.572924 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.574316 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.574722 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.574834 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.575253 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.575810 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.575942 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.576044 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.576123 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6500,"thread_start_us":6409,"threads_started":1}
I20260925 01:23:56.593237 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.586845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:53639 (local address 127.6.236.129:37589)
0925 01:23:56.587110 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.587117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.587885 (+ 768us) server_negotiation.cc:407] Connection header received
0925 01:23:56.588082 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.588085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.588149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.588221 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:56.588782 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.589609 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.591619 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.592060 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.592151 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.592617 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.592727 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.592789 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.592935 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.592998 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:23:56.593394 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.586826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.6.236.130:53639)
0925 01:23:56.587788 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.587806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.587942 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.588256 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.588261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.588274 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:56.588625 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.588636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.589731 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.589737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.591490 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.591506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.592217 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.592224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.592392 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.592458 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.592812 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.592824 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.592896 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.593031 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":638,"threads_started":1}
I20260925 01:23:56.638679 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.631956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:51675 (local address 127.6.236.130:37413)
0925 01:23:56.632126 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.632132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.632751 (+ 619us) server_negotiation.cc:407] Connection header received
0925 01:23:56.632927 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.632931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.633035 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.633126 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:56.633695 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.634634 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.636753 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.637317 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.637432 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.637932 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.638059 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.638115 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.638339 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.638412 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:23:56.639384 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.632029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37413 (local address 127.6.236.129:51675)
0925 01:23:56.632643 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:56.632671 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:56.632814 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:56.633149 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:56.633154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:56.633166 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:56.633514 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.633526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.634744 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.634749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.636565 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:56.636580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.637460 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.637466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:56.637673 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.637740 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:56.638189 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:56.638198 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:23:56.638856 (+ 658us) client_negotiation.cc:240] Negotiation successful
0925 01:23:56.638995 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260925 01:23:56.684670 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:56.684902 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:56.688575 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Sending DeleteTablet for 3 replicas of tablet d468d14c1f834b9e9054b2f73c1a0d2b
I20260925 01:23:56.689548 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:56 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60126
I20260925 01:23:56.690137 14941 tablet_service.cc:1514] Processing DeleteTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:56 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260925 01:23:56.691040 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet d468d14c1f834b9e9054b2f73c1a0d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:56 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54316
I20260925 01:23:56.694703 15248 tablet_replica.cc:330] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: stopping tablet replica
I20260925 01:23:56.694962 15248 raft_consensus.cc:2229] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:56.695284 15248 raft_consensus.cc:2258] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.696476 15248 ts_tablet_manager.cc:1905] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.704576 15249 tablet_replica.cc:330] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: stopping tablet replica
I20260925 01:23:56.705297 15249 raft_consensus.cc:2229] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:56.706184 15249 raft_consensus.cc:2258] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.708050 15249 ts_tablet_manager.cc:1905] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.708792 15250 tablet_replica.cc:330] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: stopping tablet replica
I20260925 01:23:56.708997 15250 raft_consensus.cc:2229] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:56.713547 15248 ts_tablet_manager.cc:1918] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:56.713754 15248 log.cc:1196] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/d468d14c1f834b9e9054b2f73c1a0d2b
I20260925 01:23:56.714359 15248 ts_tablet_manager.cc:1939] T d468d14c1f834b9e9054b2f73c1a0d2b P f66afdd71e2b4ab3b1781109bd897431: Deleting consensus metadata
I20260925 01:23:56.715600 14651 catalog_manager.cc:4841] TS f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589): tablet d468d14c1f834b9e9054b2f73c1a0d2b (table test-table [id=41ef4ba927f54b5988332ee3910273aa]) successfully deleted
I20260925 01:23:56.709251 15250 raft_consensus.cc:2258] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:56.719432 15250 ts_tablet_manager.cc:1905] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:56.723912 15249 ts_tablet_manager.cc:1918] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:56.724102 15249 log.cc:1196] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/d468d14c1f834b9e9054b2f73c1a0d2b
I20260925 01:23:56.724602 15249 ts_tablet_manager.cc:1939] T d468d14c1f834b9e9054b2f73c1a0d2b P 67220d865fe7492f845a84569ee20624: Deleting consensus metadata
I20260925 01:23:56.725724 14653 catalog_manager.cc:4841] TS 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): tablet d468d14c1f834b9e9054b2f73c1a0d2b (table test-table [id=41ef4ba927f54b5988332ee3910273aa]) successfully deleted
I20260925 01:23:56.726614 15250 ts_tablet_manager.cc:1918] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:56.726812 15250 log.cc:1196] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/d468d14c1f834b9e9054b2f73c1a0d2b
I20260925 01:23:56.727265 15250 ts_tablet_manager.cc:1939] T d468d14c1f834b9e9054b2f73c1a0d2b P dc60764d45484c07987b04de1c08901f: Deleting consensus metadata
I20260925 01:23:56.728364 14650 catalog_manager.cc:4841] TS dc60764d45484c07987b04de1c08901f (127.6.236.131:44835): tablet d468d14c1f834b9e9054b2f73c1a0d2b (table test-table [id=41ef4ba927f54b5988332ee3910273aa]) successfully deleted
Sep 25 01:23:56 dist-test-slave-s0b2 krb5kdc[14621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:23:56 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:23:56 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:23:56.779579 15258 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:23:56.754976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.6.236.190:43867)
0925 01:23:56.766971 (+ 11995us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.766978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.767001 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:23:56.767075 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.767080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.767174 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.767303 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:56.768968 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.770151 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.771991 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.772640 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.772764 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.779256 (+ 6492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60786: BlockingRecv error: recv got EOF from 127.0.0.1:60786 (error 108)
Metrics: {"server-negotiator.queue_time_us":11852,"thread_start_us":11742,"threads_started":1}
Sep 25 01:23:56 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM
I20260925 01:23:56.815317 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:56.783815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.6.236.190:43867)
0925 01:23:56.783980 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:56.783985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:56.786729 (+ 2744us) server_negotiation.cc:407] Connection header received
0925 01:23:56.786894 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:56.786899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:56.786981 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:56.787085 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:56.788635 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.789761 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.791597 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:56.792257 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:56.792386 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:56.799905 (+ 7519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:56.799912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:56.799917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:56.799960 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:56.812009 (+ 12049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:56.812951 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:56.812957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:56.812961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:56.813200 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:56.813660 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:56.813665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:56.813669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:56.814040 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:56.814353 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:56.814697 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 01:23:56.814937 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:23:56.819346 14665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:56.820060 14665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:56.827803 15082 tablet_service.cc:1467] Processing CreateTablet for tablet d75d8150fc88494a93c32c6fe4531b9d (DEFAULT_TABLE table=test-table [id=660fad941a604b61be03e14371fa8d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.827807 14803 tablet_service.cc:1467] Processing CreateTablet for tablet d75d8150fc88494a93c32c6fe4531b9d (DEFAULT_TABLE table=test-table [id=660fad941a604b61be03e14371fa8d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.828251 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d8150fc88494a93c32c6fe4531b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.828311 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d8150fc88494a93c32c6fe4531b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.831564 14941 tablet_service.cc:1467] Processing CreateTablet for tablet d75d8150fc88494a93c32c6fe4531b9d (DEFAULT_TABLE table=test-table [id=660fad941a604b61be03e14371fa8d20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:56.832029 14941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75d8150fc88494a93c32c6fe4531b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:56.835029 15264 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Bootstrap starting.
I20260925 01:23:56.836303 15264 tablet_bootstrap.cc:654] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.840938 15266 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Bootstrap starting.
I20260925 01:23:56.842952 15266 tablet_bootstrap.cc:654] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.845219 15264 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: No bootstrap required, opened a new log
I20260925 01:23:56.845418 15264 ts_tablet_manager.cc:1397] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260925 01:23:56.845567 15266 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: No bootstrap required, opened a new log
I20260925 01:23:56.845758 15266 ts_tablet_manager.cc:1397] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 01:23:56.846351 15264 raft_consensus.cc:348] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.846673 15264 raft_consensus.cc:374] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.846568 15266 raft_consensus.cc:348] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.846853 15266 raft_consensus.cc:374] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.846844 15264 raft_consensus.cc:729] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60764d45484c07987b04de1c08901f, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.846976 15266 raft_consensus.cc:729] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67220d865fe7492f845a84569ee20624, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.847275 15266 consensus_queue.cc:260] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.847846 15266 ts_tablet_manager.cc:1428] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:56.849792 15264 consensus_queue.cc:260] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.850464 15264 ts_tablet_manager.cc:1428] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 01:23:56.851110 15268 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Bootstrap starting.
I20260925 01:23:56.854187 15268 tablet_bootstrap.cc:654] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:56.857893 15268 tablet_bootstrap.cc:492] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: No bootstrap required, opened a new log
I20260925 01:23:56.858057 15268 ts_tablet_manager.cc:1397] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:23:56.858964 15268 raft_consensus.cc:348] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.859251 15268 raft_consensus.cc:374] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:56.859385 15268 raft_consensus.cc:729] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Initialized, Role: FOLLOWER
I20260925 01:23:56.859639 15268 consensus_queue.cc:260] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:56.860255 15268 ts_tablet_manager.cc:1428] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:57.018816 15180 raft_consensus.cc:482] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:57.019033 15180 raft_consensus.cc:504] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:57.020535 15180 leader_election.cc:288] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:57.027060 15241 raft_consensus.cc:482] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:57.027297 15241 raft_consensus.cc:504] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:57.028208 15241 leader_election.cc:288] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:57.028699 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.021024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:51725 (local address 127.6.236.130:37413)
0925 01:23:57.021360 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.021366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.022159 (+ 793us) server_negotiation.cc:407] Connection header received
0925 01:23:57.022445 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.022454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.022528 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.022625 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:57.023343 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.024350 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.026169 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.027515 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.027699 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.027780 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.028259 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.028401 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260925 01:23:57.028808 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f" is_pre_election: true
I20260925 01:23:57.029098 15106 raft_consensus.cc:2454] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 0.
I20260925 01:23:57.029116 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431" is_pre_election: true
I20260925 01:23:57.029368 14823 raft_consensus.cc:2454] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 0.
I20260925 01:23:57.029521 14893 leader_election.cc:302] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, dc60764d45484c07987b04de1c08901f; no voters:
I20260925 01:23:57.029793 15242 raft_consensus.cc:2781] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:57.029881 15242 raft_consensus.cc:482] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:57.029966 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.021272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37413 (local address 127.6.236.131:51725)
0925 01:23:57.022053 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:57.022072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:57.022294 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:57.022670 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:57.022675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:57.022706 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:57.023172 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.023186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.024468 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.024472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.026014 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.026026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.027899 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.027905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.028051 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.028083 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:23:57.028860 (+ 777us) client_negotiation.cc:240] Negotiation successful
0925 01:23:57.028994 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":6,"thread_start_us":684,"threads_started":1}
I20260925 01:23:57.029982 15242 raft_consensus.cc:3037] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:57.031028 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "dc60764d45484c07987b04de1c08901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624" is_pre_election: true
I20260925 01:23:57.032405 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.021048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:56861 (local address 127.6.236.129:37589)
0925 01:23:57.021250 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.021256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.021838 (+ 582us) server_negotiation.cc:407] Connection header received
0925 01:23:57.022151 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.022166 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.022230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.022323 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:23:57.023048 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.024056 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.026473 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.027630 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.027781 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.027844 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.032022 (+ 4178us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.032171 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:23:57.032445 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.021011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.6.236.131:56861)
0925 01:23:57.021717 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:57.021737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:57.021960 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:57.022519 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:57.022523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:57.022548 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:23:57.022877 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.022888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.024211 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.024217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.026314 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.026330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.030961 (+ 4631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.030977 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.031817 (+ 840us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.031853 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:23:57.031947 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:23:57.032094 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":4,"thread_start_us":386,"threads_started":1}
I20260925 01:23:57.033087 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "dc60764d45484c07987b04de1c08901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431" is_pre_election: true
I20260925 01:23:57.033138 15242 raft_consensus.cc:504] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:57.033321 14823 raft_consensus.cc:2454] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc60764d45484c07987b04de1c08901f in term 0.
I20260925 01:23:57.033656 14962 raft_consensus.cc:2379] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc60764d45484c07987b04de1c08901f in current term 1: Already voted for candidate 67220d865fe7492f845a84569ee20624 in this term.
I20260925 01:23:57.033826 15033 leader_election.cc:302] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:57.034166 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431"
I20260925 01:23:57.034263 15180 raft_consensus.cc:2781] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:57.034375 15180 raft_consensus.cc:482] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:57.034401 14823 raft_consensus.cc:3037] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:57.034466 15180 raft_consensus.cc:3037] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:57.034723 15242 leader_election.cc:288] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 election: Requested vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), dc60764d45484c07987b04de1c08901f (127.6.236.131:44835)
I20260925 01:23:57.036067 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "67220d865fe7492f845a84569ee20624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f"
I20260925 01:23:57.036945 14823 raft_consensus.cc:2454] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67220d865fe7492f845a84569ee20624 in term 1.
I20260925 01:23:57.037298 14894 leader_election.cc:302] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67220d865fe7492f845a84569ee20624, f66afdd71e2b4ab3b1781109bd897431; no voters:
I20260925 01:23:57.037567 15242 raft_consensus.cc:2781] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:57.037477 15180 raft_consensus.cc:504] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:57.037804 15242 raft_consensus.cc:686] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 LEADER]: Becoming Leader. State: Replica: 67220d865fe7492f845a84569ee20624, State: Running, Role: LEADER
I20260925 01:23:57.038113 15242 consensus_queue.cc:237] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } }
I20260925 01:23:57.038425 15106 raft_consensus.cc:2379] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67220d865fe7492f845a84569ee20624 in current term 1: Already voted for candidate dc60764d45484c07987b04de1c08901f in this term.
I20260925 01:23:57.038805 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "dc60764d45484c07987b04de1c08901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624"
I20260925 01:23:57.039474 15180 leader_election.cc:288] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [CANDIDATE]: Term 1 election: Requested vote from peers f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:57.039499 14823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75d8150fc88494a93c32c6fe4531b9d" candidate_uuid: "dc60764d45484c07987b04de1c08901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66afdd71e2b4ab3b1781109bd897431"
I20260925 01:23:57.039769 14823 raft_consensus.cc:2379] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dc60764d45484c07987b04de1c08901f in current term 1: Already voted for candidate 67220d865fe7492f845a84569ee20624 in this term.
I20260925 01:23:57.040194 15033 leader_election.cc:302] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc60764d45484c07987b04de1c08901f; no voters: 67220d865fe7492f845a84569ee20624, f66afdd71e2b4ab3b1781109bd897431
I20260925 01:23:57.040491 15180 raft_consensus.cc:2726] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 01:23:57.042412 14666 catalog_manager.cc:5495] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67220d865fe7492f845a84569ee20624 (127.6.236.130). New cstate: current_term: 1 leader_uuid: "67220d865fe7492f845a84569ee20624" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:57.086861 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.081403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.6.236.130:37413)
0925 01:23:57.081587 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.081593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.081790 (+ 197us) server_negotiation.cc:407] Connection header received
0925 01:23:57.081979 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.081983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.082056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.082135 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:57.082521 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.083404 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.085025 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.085566 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.085694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.086090 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:57.086256 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:57.086347 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.086468 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.086554 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:23:57.089418 15106 raft_consensus.cc:1261] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Refusing update from remote peer 67220d865fe7492f845a84569ee20624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:57.089498 14823 raft_consensus.cc:1261] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Refusing update from remote peer 67220d865fe7492f845a84569ee20624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:57.090090 15241 consensus_queue.cc:1035] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:57.090363 15242 consensus_queue.cc:1035] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:57.102144 14666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:57.102407 14666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:57.108476 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d8150fc88494a93c32c6fe4531b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54316
I20260925 01:23:57.108757 15250 tablet_replica.cc:330] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: stopping tablet replica
I20260925 01:23:57.108880 14941 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d8150fc88494a93c32c6fe4531b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260925 01:23:57.108956 15250 raft_consensus.cc:2229] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:57.109186 15249 tablet_replica.cc:330] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: stopping tablet replica
I20260925 01:23:57.109170 15250 raft_consensus.cc:2258] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.109344 15249 raft_consensus.cc:2229] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:57.109635 15249 raft_consensus.cc:2258] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.108590 14666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Sending DeleteTablet for 3 replicas of tablet d75d8150fc88494a93c32c6fe4531b9d
I20260925 01:23:57.110585 14941 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d8150fc88494a93c32c6fe4531b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260925 01:23:57.110790 15249 ts_tablet_manager.cc:1905] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.110757 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d8150fc88494a93c32c6fe4531b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60126
I20260925 01:23:57.111009 15248 tablet_replica.cc:330] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: stopping tablet replica
I20260925 01:23:57.111184 15248 raft_consensus.cc:2229] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:57.111394 15248 raft_consensus.cc:2258] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.113097 15250 ts_tablet_manager.cc:1905] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.113112 15248 ts_tablet_manager.cc:1905] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.114730 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet d75d8150fc88494a93c32c6fe4531b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54316
I20260925 01:23:57.118059 15249 ts_tablet_manager.cc:1918] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:57.118261 15249 log.cc:1196] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/d75d8150fc88494a93c32c6fe4531b9d
I20260925 01:23:57.118723 15249 ts_tablet_manager.cc:1939] T d75d8150fc88494a93c32c6fe4531b9d P 67220d865fe7492f845a84569ee20624: Deleting consensus metadata
I20260925 01:23:57.118522 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:23:57.119163 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:23:57.119195 15248 ts_tablet_manager.cc:1918] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:57.119323 15248 log.cc:1196] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/d75d8150fc88494a93c32c6fe4531b9d
I20260925 01:23:57.119787 15248 ts_tablet_manager.cc:1939] T d75d8150fc88494a93c32c6fe4531b9d P f66afdd71e2b4ab3b1781109bd897431: Deleting consensus metadata
I20260925 01:23:57.121831 15250 ts_tablet_manager.cc:1918] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:23:57.121975 15250 log.cc:1196] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/d75d8150fc88494a93c32c6fe4531b9d
I20260925 01:23:57.120787 14651 catalog_manager.cc:4841] TS f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589): tablet d75d8150fc88494a93c32c6fe4531b9d (table test-table [id=660fad941a604b61be03e14371fa8d20]) successfully deleted
I20260925 01:23:57.122383 15250 ts_tablet_manager.cc:1939] T d75d8150fc88494a93c32c6fe4531b9d P dc60764d45484c07987b04de1c08901f: Deleting consensus metadata
W20260925 01:23:57.120806 14653 catalog_manager.cc:4808] TS 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): delete failed for tablet d75d8150fc88494a93c32c6fe4531b9d because the tablet was not found. No further retry: Not found: Tablet not found: d75d8150fc88494a93c32c6fe4531b9d
W20260925 01:23:57.125077 14650 catalog_manager.cc:4808] TS dc60764d45484c07987b04de1c08901f (127.6.236.131:44835): delete failed for tablet d75d8150fc88494a93c32c6fe4531b9d because the tablet was not found. No further retry: Not found: Tablet not found: d75d8150fc88494a93c32c6fe4531b9d
I20260925 01:23:57.128576 15082 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b (DEFAULT_TABLE table=test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:57.128801 14803 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b (DEFAULT_TABLE table=test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:57.128958 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5892e24f14cb788ca9d20f92dc73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:57.129168 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5892e24f14cb788ca9d20f92dc73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:57.129087 14941 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b (DEFAULT_TABLE table=test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:23:57.129508 14941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5892e24f14cb788ca9d20f92dc73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:23:57.134958 15268 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Bootstrap starting.
I20260925 01:23:57.135003 15264 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Bootstrap starting.
I20260925 01:23:57.136451 15268 tablet_bootstrap.cc:654] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:57.136461 15264 tablet_bootstrap.cc:654] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:57.137461 15266 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Bootstrap starting.
I20260925 01:23:57.138906 15264 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: No bootstrap required, opened a new log
I20260925 01:23:57.139050 15264 ts_tablet_manager.cc:1397] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:23:57.139106 15266 tablet_bootstrap.cc:654] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Neither blocks nor log segments found. Creating new log.
I20260925 01:23:57.140161 15268 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: No bootstrap required, opened a new log
I20260925 01:23:57.140105 15264 raft_consensus.cc:348] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.140318 15268 ts_tablet_manager.cc:1397] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:23:57.140334 15264 raft_consensus.cc:374] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:57.140403 15264 raft_consensus.cc:729] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc60764d45484c07987b04de1c08901f, State: Initialized, Role: FOLLOWER
I20260925 01:23:57.140614 15264 consensus_queue.cc:260] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.141085 15264 ts_tablet_manager.cc:1428] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:23:57.140995 15268 raft_consensus.cc:348] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.141249 15268 raft_consensus.cc:374] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:57.141316 15268 raft_consensus.cc:729] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Initialized, Role: FOLLOWER
I20260925 01:23:57.141544 15268 consensus_queue.cc:260] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.141799 15187 raft_consensus.cc:482] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:23:57.141947 15187 raft_consensus.cc:504] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.142570 15187 leader_election.cc:288] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc60764d45484c07987b04de1c08901f (127.6.236.131:44835), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:57.143213 15268 ts_tablet_manager.cc:1428] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:23:57.143596 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5892e24f14cb788ca9d20f92dc73b" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624" is_pre_election: true
I20260925 01:23:57.144390 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5892e24f14cb788ca9d20f92dc73b" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f" is_pre_election: true
W20260925 01:23:57.144479 14756 leader_election.cc:341] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): Illegal state: must be running to vote when last-logged opid is not known
I20260925 01:23:57.144604 15106 raft_consensus.cc:2454] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 0.
I20260925 01:23:57.145007 14753 leader_election.cc:302] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters: 67220d865fe7492f845a84569ee20624
I20260925 01:23:57.145476 15187 raft_consensus.cc:2781] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:23:57.145577 15187 raft_consensus.cc:482] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:23:57.145666 15187 raft_consensus.cc:3037] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:57.148581 15187 raft_consensus.cc:504] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.149816 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5892e24f14cb788ca9d20f92dc73b" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67220d865fe7492f845a84569ee20624"
I20260925 01:23:57.149961 15106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5892e24f14cb788ca9d20f92dc73b" candidate_uuid: "f66afdd71e2b4ab3b1781109bd897431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc60764d45484c07987b04de1c08901f"
I20260925 01:23:57.150143 15106 raft_consensus.cc:3037] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 0 FOLLOWER]: Advancing to term 1
W20260925 01:23:57.150439 14756 leader_election.cc:341] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): Illegal state: must be running to vote when last-logged opid is not known
I20260925 01:23:57.151446 15187 leader_election.cc:288] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Requested vote from peers dc60764d45484c07987b04de1c08901f (127.6.236.131:44835), 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413)
I20260925 01:23:57.153020 15106 raft_consensus.cc:2454] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66afdd71e2b4ab3b1781109bd897431 in term 1.
I20260925 01:23:57.153455 14753 leader_election.cc:302] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc60764d45484c07987b04de1c08901f, f66afdd71e2b4ab3b1781109bd897431; no voters: 67220d865fe7492f845a84569ee20624
I20260925 01:23:57.155073 15187 raft_consensus.cc:2781] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:23:57.155346 15187 raft_consensus.cc:686] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 1 LEADER]: Becoming Leader. State: Replica: f66afdd71e2b4ab3b1781109bd897431, State: Running, Role: LEADER
I20260925 01:23:57.155642 15187 consensus_queue.cc:237] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.158167 14665 catalog_manager.cc:5495] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 reported cstate change: term changed from 0 to 1, leader changed from <none> to f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "f66afdd71e2b4ab3b1781109bd897431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } health_report { overall_health: UNKNOWN } } }
I20260925 01:23:57.164181 15266 tablet_bootstrap.cc:492] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: No bootstrap required, opened a new log
I20260925 01:23:57.164410 15266 ts_tablet_manager.cc:1397] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.014s
I20260925 01:23:57.165241 15266 raft_consensus.cc:348] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.165584 15266 raft_consensus.cc:374] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:23:57.165702 15266 raft_consensus.cc:729] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67220d865fe7492f845a84569ee20624, State: Initialized, Role: FOLLOWER
I20260925 01:23:57.166018 15266 consensus_queue.cc:260] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 } } peers { permanent_uuid: "f66afdd71e2b4ab3b1781109bd897431" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 37589 } } peers { permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 } }
I20260925 01:23:57.166617 15266 ts_tablet_manager.cc:1428] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 01:23:57.185729 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.179602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.6.236.129:37589)
0925 01:23:57.179752 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.179757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.179774 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:23:57.179845 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.179850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.179922 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.179995 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:57.180512 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.181641 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.183611 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.184259 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.184365 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.184972 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:57.185103 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:57.185178 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.185360 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.185440 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 01:23:57.202276 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.197932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:44139 (local address 127.6.236.129:37589)
0925 01:23:57.198110 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.198114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.198465 (+ 351us) server_negotiation.cc:407] Connection header received
0925 01:23:57.198601 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.198604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.198663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.198730 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:23:57.199061 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.199704 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.200922 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.201339 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.201427 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.201736 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:23:57.201824 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:23:57.201884 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.201964 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.202019 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:23:57.202340 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.197831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:37589 (local address 127.6.236.129:44139)
0925 01:23:57.198386 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 01:23:57.198402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:23:57.198503 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:23:57.198744 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:23:57.198747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:23:57.198756 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:23:57.198951 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.198958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.199800 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.199803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.200784 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:23:57.200795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.201471 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.201475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:23:57.201583 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.201617 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:23:57.201891 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:23:57.201897 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:23:57.201950 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 01:23:57.202043 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":281,"threads_started":1}
I20260925 01:23:57.209282 14962 raft_consensus.cc:3037] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:23:57.209383 15106 raft_consensus.cc:1261] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:57.209892 15229 consensus_queue.cc:1035] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc60764d45484c07987b04de1c08901f" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:57.212291 14962 raft_consensus.cc:1261] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Refusing update from remote peer f66afdd71e2b4ab3b1781109bd897431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:23:57.212838 15229 consensus_queue.cc:1035] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67220d865fe7492f845a84569ee20624" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:23:57.253847 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:57.254083 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:23:57.256664 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd3da72e917643f298dac0b4581bd335: Sending DeleteTablet for 3 replicas of tablet a8b5892e24f14cb788ca9d20f92dc73b
I20260925 01:23:57.257325 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54316
I20260925 01:23:57.257568 14941 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:41368
I20260925 01:23:57.257628 15250 tablet_replica.cc:330] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: stopping tablet replica
I20260925 01:23:57.257805 15250 raft_consensus.cc:2229] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:57.257856 15249 tablet_replica.cc:330] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: stopping tablet replica
I20260925 01:23:57.257997 15250 raft_consensus.cc:2258] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.258016 15249 raft_consensus.cc:2229] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:23:57.258486 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14737
I20260925 01:23:57.258189 15249 raft_consensus.cc:2258] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.259578 15250 ts_tablet_manager.cc:1905] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.259793 15249 ts_tablet_manager.cc:1905] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.259819 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b5892e24f14cb788ca9d20f92dc73b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:23:57 UTC) from {username='oryx', principal='oryx/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60126
I20260925 01:23:57.260067 15248 tablet_replica.cc:330] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: stopping tablet replica
I20260925 01:23:57.260237 15248 raft_consensus.cc:2229] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:23:57.260551 15248 raft_consensus.cc:2258] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:23:57.261672 15248 ts_tablet_manager.cc:1905] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:23:57.266090 15249 ts_tablet_manager.cc:1918] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:57.266089 15250 ts_tablet_manager.cc:1918] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:57.266229 15250 log.cc:1196] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/a8b5892e24f14cb788ca9d20f92dc73b
I20260925 01:23:57.266376 15249 log.cc:1196] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/a8b5892e24f14cb788ca9d20f92dc73b
I20260925 01:23:57.266713 15250 ts_tablet_manager.cc:1939] T a8b5892e24f14cb788ca9d20f92dc73b P dc60764d45484c07987b04de1c08901f: Deleting consensus metadata
I20260925 01:23:57.266812 15249 ts_tablet_manager.cc:1939] T a8b5892e24f14cb788ca9d20f92dc73b P 67220d865fe7492f845a84569ee20624: Deleting consensus metadata
I20260925 01:23:57.268147 14653 catalog_manager.cc:4841] TS 67220d865fe7492f845a84569ee20624 (127.6.236.130:37413): tablet a8b5892e24f14cb788ca9d20f92dc73b (table test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]) successfully deleted
I20260925 01:23:57.268424 15248 ts_tablet_manager.cc:1918] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:23:57.268586 15248 log.cc:1196] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/a8b5892e24f14cb788ca9d20f92dc73b
I20260925 01:23:57.268993 15248 ts_tablet_manager.cc:1939] T a8b5892e24f14cb788ca9d20f92dc73b P f66afdd71e2b4ab3b1781109bd897431: Deleting consensus metadata
I20260925 01:23:57.270016 14651 catalog_manager.cc:4841] TS f66afdd71e2b4ab3b1781109bd897431 (127.6.236.129:37589): tablet a8b5892e24f14cb788ca9d20f92dc73b (table test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]) successfully deleted
I20260925 01:23:57.270036 14650 catalog_manager.cc:4841] TS dc60764d45484c07987b04de1c08901f (127.6.236.131:44835): tablet a8b5892e24f14cb788ca9d20f92dc73b (table test-table [id=d8ce0c82b42849ca83a8ccbb5400b742]) successfully deleted
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.129@KRBTEST.COM
I20260925 01:23:57.276194 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.259765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.6.236.129:37589)
0925 01:23:57.259929 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.259935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.260487 (+ 552us) server_negotiation.cc:407] Connection header received
0925 01:23:57.260738 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.260743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.260826 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.260907 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:57.262585 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.263676 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.265682 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.266349 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.266445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.271267 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:57.271281 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:57.271297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:57.271328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:57.273921 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.274531 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.274538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.274540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.274676 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.275062 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.275065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.275067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.275381 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:57.275600 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.275806 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.275963 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:23:57.276845 14868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:57.420028 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14737
I20260925 01:23:57.443266 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14877
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.130@KRBTEST.COM
I20260925 01:23:57.457311 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.444069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.6.236.130:37413)
0925 01:23:57.444257 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.444264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.444397 (+ 133us) server_negotiation.cc:407] Connection header received
0925 01:23:57.444553 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.444557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.444617 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.444702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:57.445796 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.446532 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.447675 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.448035 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.448118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.451766 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:57.451779 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:57.451794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:57.451833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:57.454457 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.455285 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.455293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.455295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.455458 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.455811 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.455817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.455820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.456182 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:57.456468 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.456789 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.457003 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:23:57.457952 15007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:57.578516 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14877
I20260925 01:23:57.597343 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15016
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[14621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.236.131@KRBTEST.COM
I20260925 01:23:57.611718 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:23:57.598058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.6.236.131:44835)
0925 01:23:57.598527 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 01:23:57.598532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:23:57.598546 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:23:57.598597 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:23:57.598601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:23:57.598676 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:23:57.598767 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:23:57.600044 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.600738 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.601930 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:23:57.602366 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:23:57.602470 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:23:57.606552 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:23:57.606567 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:23:57.606584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:23:57.606623 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:23:57.609302 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.609915 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.609922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.609925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.610076 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:23:57.610513 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:23:57.610516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:23:57.610518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:23:57.610850 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:23:57.611073 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 01:23:57.611293 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 01:23:57.611452 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260925 01:23:57.612366 15146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:57.707590 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15016
I20260925 01:23:57.723299 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 14634
I20260925 01:23:57.724411 14706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:23:57.838101 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 14634
2026-09-25T01:23:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6388 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[15295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[15295](info): set up 2 sockets
Sep 25 01:23:57 dist-test-slave-s0b2 krb5kdc[15295](info): commencing operation
krb5kdc: starting...
W20260925 01:23:58.831717 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:23:58 dist-test-slave-s0b2 krb5kdc[15295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:23:59.033077 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44049
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:44049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:23:59.345112 15308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:23:59.345388 15308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:23:59.345507 15308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:23:59.352658 15308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:23:59.352756 15308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:23:59.352820 15308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:23:59.352903 15308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:23:59.352967 15308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:23:59.362684 15308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:44049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:23:59.363911 15308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:23:59.365156 15308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:23:59.376030 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:59.376091 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:23:59.376139 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:23:59.376943 15308 server_base.cc:1034] running on GCE node
I20260925 01:23:59.377714 15308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:23:59.379122 15308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:23:59.380311 15308 hybrid_clock.cc:648] HybridClock initialized: now 1790299439380254 us; error 48 us; skew 500 ppm
I20260925 01:23:59.380641 15308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T01:23:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1614 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:23:59 dist-test-slave-s0b2 krb5kdc[15322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:23:59 dist-test-slave-s0b2 krb5kdc[15322](info): set up 2 sockets
Sep 25 01:23:59 dist-test-slave-s0b2 krb5kdc[15322](info): commencing operation
krb5kdc: starting...
W20260925 01:24:00.447737 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:24:00 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:24:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:24:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:24:00.649358 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40991
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:00.958943 15335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:00.959215 15335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:00.959347 15335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:00.967376 15335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:24:00.967499 15335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:00.967574 15335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:00.967651 15335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:24:00.967710 15335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:24:00.977293 15335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:00.978587 15335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:00.979866 15335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:00.990666 15340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:00.990705 15341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:00.990726 15343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:00.991506 15335 server_base.cc:1034] running on GCE node
I20260925 01:24:00.992203 15335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:00.993506 15335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:00.994694 15335 hybrid_clock.cc:648] HybridClock initialized: now 1790299440994635 us; error 42 us; skew 500 ppm
I20260925 01:24:00.995004 15335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:00 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:00.997888 15335 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:24:00.999289 15335 webserver.cc:460] Webserver started at http://127.6.236.190:44473/ using document root <none> and password file <none>
I20260925 01:24:00.999948 15335 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:01.000059 15335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:01.000345 15335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:01.003285 15335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "faeead62711d4c32a451b162923567d4"
format_stamp: "Formatted at 2026-09-25 01:24:00 on dist-test-slave-s0b2"
server_key: "0bf03ea3db8d09f254192e4d3222c420"
server_key_iv: "00f63f5e134e4f84273951f61ccc8f4e"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.004024 15335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "faeead62711d4c32a451b162923567d4"
format_stamp: "Formatted at 2026-09-25 01:24:00 on dist-test-slave-s0b2"
server_key: "0bf03ea3db8d09f254192e4d3222c420"
server_key_iv: "00f63f5e134e4f84273951f61ccc8f4e"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.009157 15335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:24:01.012749 15350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:01.014010 15335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:01.014187 15335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "faeead62711d4c32a451b162923567d4"
format_stamp: "Formatted at 2026-09-25 01:24:00 on dist-test-slave-s0b2"
server_key: "0bf03ea3db8d09f254192e4d3222c420"
server_key_iv: "00f63f5e134e4f84273951f61ccc8f4e"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.014430 15335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:01.029444 15335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:01.033252 15335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:01.033505 15335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:01.049590 15335 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:40991
I20260925 01:24:01.049626 15401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:40991 every 8 connection(s)
I20260925 01:24:01.051024 15335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:24:01.054669 15402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:01.056829 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15335
I20260925 01:24:01.057016 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:24:01.057649 7090 external_mini_cluster.cc:1428] Setting key 21da1489f1a723d87e3304671808ee0a
I20260925 01:24:01.066020 15402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Bootstrap starting.
I20260925 01:24:01.069621 15402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:01.070883 15402 log.cc:824] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:24:01 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:01.073951 15402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: No bootstrap required, opened a new log
I20260925 01:24:01.079816 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:01.059675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.6.236.190:40991)
0925 01:24:01.060392 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:01.060404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:01.060444 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:24:01.062107 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:01.062129 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:01.062571 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:01.062902 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:01.064610 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.065859 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.067556 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.068209 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.068330 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:01.072945 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:01.072952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:01.072966 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:01.072998 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:01.075925 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:01.076687 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:01.076693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:01.076699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:01.076857 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:01.077233 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:01.077236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:01.077238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:01.077751 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:01.077963 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:01.078443 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 01:24:01.078825 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":320,"threads_started":1}
I20260925 01:24:01.082796 15402 raft_consensus.cc:348] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:01.083122 15402 raft_consensus.cc:374] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:01.083209 15402 raft_consensus.cc:729] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faeead62711d4c32a451b162923567d4, State: Initialized, Role: FOLLOWER
I20260925 01:24:01.083702 15402 consensus_queue.cc:260] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:01.083921 15402 raft_consensus.cc:388] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:24:01.084043 15402 raft_consensus.cc:482] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:24:01.084188 15402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:01.086167 15402 raft_consensus.cc:504] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:01.086674 15402 leader_election.cc:302] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faeead62711d4c32a451b162923567d4; no voters:
I20260925 01:24:01.087175 15402 leader_election.cc:288] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:24:01.087288 15407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:24:01.087891 15407 raft_consensus.cc:686] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 LEADER]: Becoming Leader. State: Replica: faeead62711d4c32a451b162923567d4, State: Running, Role: LEADER
I20260925 01:24:01.088438 15407 consensus_queue.cc:237] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:01.089169 15402 sys_catalog.cc:564] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:24:01.090972 15408 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faeead62711d4c32a451b162923567d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } } }
I20260925 01:24:01.090981 15409 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faeead62711d4c32a451b162923567d4. Latest consensus state: current_term: 1 leader_uuid: "faeead62711d4c32a451b162923567d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } } }
I20260925 01:24:01.091375 15408 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:01.091375 15409 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:01.093072 15412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:24:01.098182 15412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:24:01.106637 15412 catalog_manager.cc:1347] Generated new cluster ID: 09e57f74cd914f399704af0bc604d115
I20260925 01:24:01.106804 15412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:24:01.124094 15412 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:24:01.125921 15412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:24:01.144660 15412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Generated new TSK 0
I20260925 01:24:01.145717 15412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:24:01.197769 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:01.510390 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:01.510667 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:01.510815 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:01.518025 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:01.518137 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:01.518321 15428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:24:01.527839 15428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:01.529054 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:01.530340 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:01.542199 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:01.542188 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:01.542135 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:01.542912 15428 server_base.cc:1034] running on GCE node
I20260925 01:24:01.543399 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:01.544222 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:01.545408 15428 hybrid_clock.cc:648] HybridClock initialized: now 1790299441545354 us; error 43 us; skew 500 ppm
I20260925 01:24:01.545663 15428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:01 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299441, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:01.548533 15428 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:24:01.549876 15428 webserver.cc:460] Webserver started at http://127.6.236.129:42645/ using document root <none> and password file <none>
I20260925 01:24:01.550539 15428 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:01.550634 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:01.550886 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:01.553576 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "be6a255d5b4b4260ad4a846dd15a6d36"
format_stamp: "Formatted at 2026-09-25 01:24:01 on dist-test-slave-s0b2"
server_key: "623b0b8c81542d9884eeb00cbb12545e"
server_key_iv: "4ef5341dbc302ba7fb16ab14653496d3"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.554288 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "be6a255d5b4b4260ad4a846dd15a6d36"
format_stamp: "Formatted at 2026-09-25 01:24:01 on dist-test-slave-s0b2"
server_key: "623b0b8c81542d9884eeb00cbb12545e"
server_key_iv: "4ef5341dbc302ba7fb16ab14653496d3"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.558997 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:24:01.562139 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:01.563373 15428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:24:01.563500 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "be6a255d5b4b4260ad4a846dd15a6d36"
format_stamp: "Formatted at 2026-09-25 01:24:01 on dist-test-slave-s0b2"
server_key: "623b0b8c81542d9884eeb00cbb12545e"
server_key_iv: "4ef5341dbc302ba7fb16ab14653496d3"
server_key_version: "encryptionkey@0"
I20260925 01:24:01.563650 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:01.583947 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:01.587597 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:01.587824 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:01.588634 15428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:01.589987 15428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:01.590060 15428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:01.590153 15428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:01.590210 15428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:01.619112 15428 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:37497
I20260925 01:24:01.619179 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:37497 every 8 connection(s)
I20260925 01:24:01.620316 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:24:01.629798 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15428
I20260925 01:24:01.629962 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:24:01.630805 7090 external_mini_cluster.cc:1428] Setting key 481121a6ab7e07b2aec49a2691387e74
Sep 25 01:24:01 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299441, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:01.639106 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:01.622646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:36151 (local address 127.6.236.190:40991)
0925 01:24:01.623065 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:01.623070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:01.623914 (+ 844us) server_negotiation.cc:407] Connection header received
0925 01:24:01.625680 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:01.625686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:01.625765 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:01.625858 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:01.627338 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.628061 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.629548 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.629987 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.630093 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:01.633933 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:01.633938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:01.633941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:01.633971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:01.636301 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:01.637056 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:01.637059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:01.637061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:01.637178 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:01.637581 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:01.637584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:01.637586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:01.637831 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:01.637934 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:01.638686 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 01:24:01.638857 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260925 01:24:01.640125 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:01.623101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:36151)
0925 01:24:01.623770 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:01.623806 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:01.625389 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:01.626094 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:01.626102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:01.626635 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:01.627173 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:01.627190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.628210 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.628213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:01.629411 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:01.629423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:01.630281 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:01.630291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:01.630600 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:01.631255 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:01.631289 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:01.633716 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:01.636427 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:01.636435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:01.636450 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:01.636965 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:01.637266 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:01.637270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:01.637274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:01.637491 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:01.637937 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:01.637944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:01.638349 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 01:24:01.638632 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 01:24:01.639105 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":242,"threads_started":1}
I20260925 01:24:01.641494 15556 heartbeater.cc:347] Connected to a master server at 127.6.236.190:40991
I20260925 01:24:01.641803 15556 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:01.642567 15556 heartbeater.cc:511] Master 127.6.236.190:40991 requested a full tablet report, sending...
I20260925 01:24:01.644838 15367 ts_manager.cc:194] Registered new tserver with Master: be6a255d5b4b4260ad4a846dd15a6d36 (127.6.236.129:37497)
I20260925 01:24:01.646288 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:36151
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:24:01.675572 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:01.982843 15562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:01.983119 15562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:01.983258 15562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:01.990404 15562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:01.990520 15562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:01.990698 15562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:24:02.000281 15562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:02.001616 15562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:02.002928 15562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:02.015316 15567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:02.015354 15568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:02.015493 15570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:02.016130 15562 server_base.cc:1034] running on GCE node
I20260925 01:24:02.016738 15562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:02.017673 15562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:02.018884 15562 hybrid_clock.cc:648] HybridClock initialized: now 1790299442018825 us; error 52 us; skew 500 ppm
I20260925 01:24:02.019181 15562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:02.022071 15562 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:24:02.023444 15562 webserver.cc:460] Webserver started at http://127.6.236.130:41975/ using document root <none> and password file <none>
I20260925 01:24:02.024075 15562 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:02.024173 15562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:02.024411 15562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:02.027205 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "e5d4bbc168c64ae78cbe5f7fae8541ab"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "c86b21594d6ac2df8870f58969036dbb"
server_key_iv: "482e98dd04fbf3d7354b7212b5b58a40"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.027901 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "e5d4bbc168c64ae78cbe5f7fae8541ab"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "c86b21594d6ac2df8870f58969036dbb"
server_key_iv: "482e98dd04fbf3d7354b7212b5b58a40"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.032799 15562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:24:02.036072 15577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.037357 15562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:02.037523 15562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "e5d4bbc168c64ae78cbe5f7fae8541ab"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "c86b21594d6ac2df8870f58969036dbb"
server_key_iv: "482e98dd04fbf3d7354b7212b5b58a40"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.037760 15562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:02.058804 15562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:02.063151 15562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:02.063416 15562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:02.064348 15562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:02.065697 15562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:02.065773 15562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.065878 15562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:02.065925 15562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.095037 15562 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43617
I20260925 01:24:02.095134 15689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43617 every 8 connection(s)
I20260925 01:24:02.096257 15562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:24:02.103271 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15562
I20260925 01:24:02.103430 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:24:02.103950 7090 external_mini_cluster.cc:1428] Setting key e2410b736740e8f5a25adfa343294791
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:02.115736 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.098733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:60553 (local address 127.6.236.190:40991)
0925 01:24:02.098905 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:02.098909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:02.100143 (+ 1234us) server_negotiation.cc:407] Connection header received
0925 01:24:02.101956 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:02.101960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:02.102021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:02.102111 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:02.103636 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.104625 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.106463 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.106868 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.106974 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.110165 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:02.110170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:02.110174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:02.110205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:02.112586 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.113432 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.113436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.113438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.113606 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.114072 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.114076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.114079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.114425 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:02.114618 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:02.115203 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 01:24:02.115418 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:24:02.116622 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.099192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:60553)
0925 01:24:02.099988 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:02.100032 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:02.101731 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:02.102330 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:02.102337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:02.102868 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:02.103450 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:02.103476 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.104794 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.104800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:02.106330 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:02.106344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.107074 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.107082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:02.107312 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.107924 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:02.107945 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:02.109954 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:02.112719 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:02.112728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:02.112747 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:02.113307 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:02.113717 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:02.113722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:02.113727 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:02.113955 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:02.114569 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:02.114575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:02.114877 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 01:24:02.115092 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 01:24:02.115468 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":306,"threads_started":1}
I20260925 01:24:02.117980 15690 heartbeater.cc:347] Connected to a master server at 127.6.236.190:40991
I20260925 01:24:02.118340 15690 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:02.119172 15690 heartbeater.cc:511] Master 127.6.236.190:40991 requested a full tablet report, sending...
I20260925 01:24:02.120604 15367 ts_manager.cc:194] Registered new tserver with Master: e5d4bbc168c64ae78cbe5f7fae8541ab (127.6.236.130:43617)
I20260925 01:24:02.121377 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:60553
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:24:02.151407 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:02.458048 15695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:02.458393 15695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:02.458547 15695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:02.465680 15695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:02.465786 15695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:02.465943 15695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:24:02.475504 15695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:02.476686 15695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:02.477931 15695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:02.489995 15701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:02.490064 15700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:02.490056 15703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:02.491174 15695 server_base.cc:1034] running on GCE node
I20260925 01:24:02.491711 15695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:02.492615 15695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:02.493808 15695 hybrid_clock.cc:648] HybridClock initialized: now 1790299442493755 us; error 47 us; skew 500 ppm
I20260925 01:24:02.494092 15695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:02.497056 15695 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:24:02.498463 15695 webserver.cc:460] Webserver started at http://127.6.236.131:37295/ using document root <none> and password file <none>
I20260925 01:24:02.499095 15695 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:02.499212 15695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:02.499488 15695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:02.502255 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "504a09bbf3734bddaf1aae6b962b59c8"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "50046553a75d87fd9f6b843d841e2f93"
server_key_iv: "0e39e8b4933879ba0aeee736f534d374"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.502998 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "504a09bbf3734bddaf1aae6b962b59c8"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "50046553a75d87fd9f6b843d841e2f93"
server_key_iv: "0e39e8b4933879ba0aeee736f534d374"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.507779 15695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:24:02.511080 15710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.512351 15695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 01:24:02.512490 15695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "504a09bbf3734bddaf1aae6b962b59c8"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "50046553a75d87fd9f6b843d841e2f93"
server_key_iv: "0e39e8b4933879ba0aeee736f534d374"
server_key_version: "encryptionkey@0"
I20260925 01:24:02.512658 15695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:02.533547 15695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:02.537341 15695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:02.537585 15695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:02.538683 15695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:02.539937 15695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:02.540014 15695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.540130 15695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:02.540192 15695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:02.569296 15695 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:34321
I20260925 01:24:02.569344 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:34321 every 8 connection(s)
I20260925 01:24:02.570549 15695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:24:02.579268 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15695
I20260925 01:24:02.579435 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:24:02.579973 7090 external_mini_cluster.cc:1428] Setting key 7a2e4f798d77add7b541ae17ae3405b9
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:02.589749 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.572985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:59837 (local address 127.6.236.190:40991)
0925 01:24:02.573160 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:02.573165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:02.574271 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 01:24:02.576045 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:02.576052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:02.576138 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:02.576225 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:02.577829 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.578659 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.580783 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.581228 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.581357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.584727 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:02.584731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:02.584735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:02.584766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:02.586967 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.587626 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.587630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.587632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.587742 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.588131 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.588136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.588138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.588398 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:02.588528 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:02.589192 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 01:24:02.589413 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:24:02.590713 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.573416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:59837)
0925 01:24:02.574093 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:02.574130 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:02.575777 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:02.576462 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:02.576472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:02.577086 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:02.577641 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:02.577662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.578815 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.578819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:02.580637 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:02.580649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.581420 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.581425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:02.581651 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.582361 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:02.582389 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:02.584528 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:02.587105 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:02.587113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:02.587129 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:02.587514 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:02.587844 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:02.587849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:02.587852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:02.588022 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:02.588550 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:02.588559 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:02.588847 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 01:24:02.589087 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 01:24:02.589588 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20260925 01:24:02.592116 15823 heartbeater.cc:347] Connected to a master server at 127.6.236.190:40991
I20260925 01:24:02.592370 15823 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:02.592998 15823 heartbeater.cc:511] Master 127.6.236.190:40991 requested a full tablet report, sending...
I20260925 01:24:02.594547 15367 ts_manager.cc:194] Registered new tserver with Master: 504a09bbf3734bddaf1aae6b962b59c8 (127.6.236.131:34321)
I20260925 01:24:02.595458 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:24:02.595455 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:59837
WARNING: no policy specified for oryx/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
Entry for principal oryx/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab.
I20260925 01:24:02.649137 15556 heartbeater.cc:503] Master 127.6.236.190:40991 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
Entry for principal oryx/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab.
WARNING: no policy specified for oryx/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
Entry for principal oryx/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab.
I20260925 01:24:02.718473 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15428
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:24:02.733768 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.719620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.6.236.129:37497)
0925 01:24:02.720284 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:02.720291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:02.720318 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:24:02.720378 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:02.720384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:02.720595 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:02.720744 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:02.721978 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.723035 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.724244 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.724788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.724877 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.728560 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:02.728571 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:02.728586 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:02.728618 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:02.731305 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.731938 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.731944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.731946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.732063 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.732364 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.732367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.732369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.732685 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:02.732902 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:02.733203 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:24:02.733430 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":339,"threads_started":1}
I20260925 01:24:02.735049 15551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:02.815430 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15428
I20260925 01:24:02.829375 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15562
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:24:02.844596 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.830276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.6.236.130:43617)
0925 01:24:02.830793 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:02.830800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:02.830826 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:24:02.830885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:02.830892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:02.831100 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:02.831235 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:02.832423 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.833397 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.834681 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.835215 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.835309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.839133 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:02.839144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:02.839158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:02.839208 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:02.841972 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.842821 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.842829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.842831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.842949 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.843279 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.843282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.843284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.843591 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:02.843809 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:02.844057 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 01:24:02.844263 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260925 01:24:02.845814 15685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:02.926185 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15562
I20260925 01:24:02.939801 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15695
Sep 25 01:24:02 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:24:02.955101 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:02.940807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.6.236.131:34321)
0925 01:24:02.941430 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:02.941437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:02.941484 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:24:02.941549 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:02.941555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:02.941767 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:02.941895 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:02.943197 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.944220 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.945432 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:02.945990 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:02.946091 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:02.950042 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:02.950053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:02.950067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:02.950104 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:02.952592 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.953173 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.953180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.953183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.953316 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:02.953592 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:02.953595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:02.953597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:02.953942 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:02.954201 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:02.954560 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 01:24:02.954758 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":310,"threads_started":1}
I20260925 01:24:02.956286 15818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:03.038348 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15695
I20260925 01:24:03.052500 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15335
I20260925 01:24:03.053634 15397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:03.154472 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15335
I20260925 01:24:03.171583 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40991
--webserver_interface=127.6.236.190
--webserver_port=44473
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:03.485694 15840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:03.485958 15840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:03.486074 15840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:03.493268 15840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:24:03.493369 15840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:03.493443 15840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:03.493517 15840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:24:03.493585 15840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:24:03.503340 15840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=44473
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:03.504581 15840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:03.505879 15840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:03.516810 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:03.516853 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:03.516891 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:03.517830 15840 server_base.cc:1034] running on GCE node
I20260925 01:24:03.518608 15840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:03.519913 15840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:03.521118 15840 hybrid_clock.cc:648] HybridClock initialized: now 1790299443521074 us; error 29 us; skew 500 ppm
I20260925 01:24:03.521426 15840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:03 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:03.524334 15840 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:24:03.525647 15840 webserver.cc:460] Webserver started at http://127.6.236.190:44473/ using document root <none> and password file <none>
I20260925 01:24:03.526315 15840 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:03.526443 15840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:03.531461 15840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:24:03.534144 15855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:03.535396 15840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:24:03.535549 15840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "faeead62711d4c32a451b162923567d4"
format_stamp: "Formatted at 2026-09-25 01:24:00 on dist-test-slave-s0b2"
server_key: "0bf03ea3db8d09f254192e4d3222c420"
server_key_iv: "00f63f5e134e4f84273951f61ccc8f4e"
server_key_version: "encryptionkey@0"
I20260925 01:24:03.536476 15840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:03.552649 15840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:03.556423 15840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:03.556668 15840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:03.573125 15840 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:40991
I20260925 01:24:03.573145 15906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:40991 every 8 connection(s)
I20260925 01:24:03.574625 15840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:24:03.580591 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15840
I20260925 01:24:03.581561 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:37497
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:40991
--webserver_port=42645
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:24:03.581826 15907 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:24:03.586076 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Bootstrap starting.
I20260925 01:24:03.605262 15907 log.cc:824] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:03.614326 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:24:03.614799 15907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Bootstrap complete.
I20260925 01:24:03.623627 15907 raft_consensus.cc:348] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:03.624428 15907 raft_consensus.cc:729] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faeead62711d4c32a451b162923567d4, State: Initialized, Role: FOLLOWER
I20260925 01:24:03.624985 15907 consensus_queue.cc:260] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:03.625198 15907 raft_consensus.cc:388] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:24:03.625300 15907 raft_consensus.cc:482] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:24:03.625458 15907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:24:03.629173 15907 raft_consensus.cc:504] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:03.629626 15907 leader_election.cc:302] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faeead62711d4c32a451b162923567d4; no voters:
I20260925 01:24:03.630215 15907 leader_election.cc:288] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:24:03.630366 15911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:24:03.631512 15911 raft_consensus.cc:686] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [term 2 LEADER]: Becoming Leader. State: Replica: faeead62711d4c32a451b162923567d4, State: Running, Role: LEADER
I20260925 01:24:03.632117 15911 consensus_queue.cc:237] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } }
I20260925 01:24:03.632355 15907 sys_catalog.cc:564] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:24:03.634482 15913 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faeead62711d4c32a451b162923567d4. Latest consensus state: current_term: 2 leader_uuid: "faeead62711d4c32a451b162923567d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } } }
I20260925 01:24:03.634815 15913 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:03.635401 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "faeead62711d4c32a451b162923567d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faeead62711d4c32a451b162923567d4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40991 } } }
I20260925 01:24:03.635694 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:03.643898 15919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:24:03.650521 15919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:24:03.652021 15919 catalog_manager.cc:1259] Loaded cluster ID: 09e57f74cd914f399704af0bc604d115
I20260925 01:24:03.652154 15919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:24:03.656193 15919 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:24:03.657704 15919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P faeead62711d4c32a451b162923567d4: Loaded TSK: 0
I20260925 01:24:03.659413 15919 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:24:03.926599 15909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:03.926910 15909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:03.927048 15909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:03.934376 15909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:03.934490 15909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:03.934638 15909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:24:03.944844 15909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.129.keytab
--principal=oryx/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:37497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=42645
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:03.946147 15909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:03.947422 15909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:03.961875 15934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:03.961892 15936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:03.962083 15933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:03.962530 15909 server_base.cc:1034] running on GCE node
I20260925 01:24:03.963116 15909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:03.963984 15909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:03.965180 15909 hybrid_clock.cc:648] HybridClock initialized: now 1790299443965124 us; error 50 us; skew 500 ppm
I20260925 01:24:03.965461 15909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:03 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299443, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:03.968464 15909 init.cc:377] Logged in from keytab as oryx/127.6.236.129@KRBTEST.COM (short username oryx)
I20260925 01:24:03.969863 15909 webserver.cc:460] Webserver started at http://127.6.236.129:42645/ using document root <none> and password file <none>
I20260925 01:24:03.970541 15909 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:03.970669 15909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:03.977010 15909 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 01:24:03.979710 15943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:03.981074 15909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:24:03.981223 15909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "be6a255d5b4b4260ad4a846dd15a6d36"
format_stamp: "Formatted at 2026-09-25 01:24:01 on dist-test-slave-s0b2"
server_key: "623b0b8c81542d9884eeb00cbb12545e"
server_key_iv: "4ef5341dbc302ba7fb16ab14653496d3"
server_key_version: "encryptionkey@0"
I20260925 01:24:03.982146 15909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:03.997934 15909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:04.001981 15909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:04.002264 15909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:04.003268 15909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:04.004688 15909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:04.004791 15909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:04.004900 15909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:04.004973 15909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:04.034821 15909 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:37497
I20260925 01:24:04.034909 16055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:37497 every 8 connection(s)
I20260925 01:24:04.036206 15909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:24:04.045478 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 15909
I20260925 01:24:04.046181 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:43617
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:40991
--webserver_port=41975
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:24:04 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:04 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:24:04.359247 16060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:04.359534 16060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:04.359671 16060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:04.366755 16060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:04.366863 16060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:04.367007 16060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:24:04.376401 16060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.130.keytab
--principal=oryx/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:43617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=41975
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:04.377763 16060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:04.379062 16060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:04.393699 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:04.393675 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:04.393914 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:04.394742 16060 server_base.cc:1034] running on GCE node
I20260925 01:24:04.395319 16060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:04.396267 16060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:04.397505 16060 hybrid_clock.cc:648] HybridClock initialized: now 1790299444397431 us; error 40 us; skew 500 ppm
I20260925 01:24:04.397795 16060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:04 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299444, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:04.404116 16060 init.cc:377] Logged in from keytab as oryx/127.6.236.130@KRBTEST.COM (short username oryx)
I20260925 01:24:04.405612 16060 webserver.cc:460] Webserver started at http://127.6.236.130:41975/ using document root <none> and password file <none>
I20260925 01:24:04.406312 16060 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:04.406438 16060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:04.412024 16060 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:24:04.414565 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:04.415907 16060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:24:04.416050 16060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "e5d4bbc168c64ae78cbe5f7fae8541ab"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "c86b21594d6ac2df8870f58969036dbb"
server_key_iv: "482e98dd04fbf3d7354b7212b5b58a40"
server_key_version: "encryptionkey@0"
I20260925 01:24:04.416941 16060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:04.435784 16060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:04.439764 16060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:04.439994 16060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:04.441045 16060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:04.442421 16060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:04.442519 16060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:04.442622 16060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:04.442687 16060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:04.484527 16060 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43617
I20260925 01:24:04.484573 16188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43617 every 8 connection(s)
I20260925 01:24:04.486022 16060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:24:04.489423 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16060
I20260925 01:24:04.490164 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:34321
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:40991
--webserver_port=37295
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:24:04 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:04 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:24:05.007745 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:05.008206 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:05.008440 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:05.021212 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:05.021378 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:05.021606 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:24:05.037024 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/krb5kdc/oryx_127.6.236.131.keytab
--principal=oryx/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:34321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=37295
--tserver_master_addrs=127.6.236.190:40991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:05.038770 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:05.040480 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:05.056838 16199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:05.057307 16200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:05.066347 16193 server_base.cc:1034] running on GCE node
W20260925 01:24:05.066805 16202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:05.067668 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:05.068974 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:05.070266 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790299445070187 us; error 43 us; skew 500 ppm
I20260925 01:24:05.070698 16193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299445, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:05.075196 16193 init.cc:377] Logged in from keytab as oryx/127.6.236.131@KRBTEST.COM (short username oryx)
I20260925 01:24:05.077111 16193 webserver.cc:460] Webserver started at http://127.6.236.131:37295/ using document root <none> and password file <none>
I20260925 01:24:05.077946 16193 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:05.078090 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:05.085282 16193 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:24:05.088600 16209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:05.090279 16193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:05.090438 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "504a09bbf3734bddaf1aae6b962b59c8"
format_stamp: "Formatted at 2026-09-25 01:24:02 on dist-test-slave-s0b2"
server_key: "50046553a75d87fd9f6b843d841e2f93"
server_key_iv: "0e39e8b4933879ba0aeee736f534d374"
server_key_version: "encryptionkey@0"
I20260925 01:24:05.091673 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:05.131414 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:05.135807 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:05.136066 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:05.137085 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:05.138496 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:05.138589 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:05.138702 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:05.138769 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260925 01:24:05.160952 16059 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.111s user 0.002s sys 0.013s
I20260925 01:24:05.163367 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:04.039185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:38979)
0925 01:24:04.039956 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:04.039991 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:04.041659 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:04.043034 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:04.043044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:04.043651 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:04.044169 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:04.044187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.046136 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.046140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:04.047427 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:04.047440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.048636 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.048640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:04.048815 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:04.049423 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:04.049444 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.162026 (+1112582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":311,"threads_started":1}
W20260925 01:24:05.163466 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:04.039060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:38979 (local address 127.6.236.190:40991)
0925 01:24:04.039845 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:04.039860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:04.040108 (+ 248us) server_negotiation.cc:407] Connection header received
0925 01:24:04.042186 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:04.042200 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:04.042564 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:04.042874 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:04.044344 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.045976 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.047570 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.048460 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.048624 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.162528 (+1113904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:38979: BlockingRecv error: recv got EOF from 127.6.236.129:38979 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":261,"threads_started":1}
W20260925 01:24:05.164381 16056 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:40991 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.236.190:40991: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:05.175529 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.165038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:42725)
0925 01:24:05.165297 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:05.165317 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:05.165428 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:05.165995 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:05.166000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:05.166480 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:05.166691 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.166703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.168108 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.168113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.169521 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.169538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.170666 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.170673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.170843 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.171447 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:05.171473 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.175257 (+ 3784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260925 01:24:05.175742 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:05.165242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42725 (local address 127.6.236.190:40991)
0925 01:24:05.165382 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.165387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.165405 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:24:05.165586 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.165592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.165675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.165750 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.166871 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.167915 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.169701 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.170311 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.170435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.175531 (+ 5096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:42725: BlockingRecv error: recv got EOF from 127.6.236.129:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:05.185503 16193 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:34321
I20260925 01:24:05.185577 16321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:34321 every 8 connection(s)
I20260925 01:24:05.185700 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.176435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:52915)
0925 01:24:05.176655 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:05.176673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:05.176766 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:05.177067 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:05.177072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:05.177438 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:05.177631 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.177642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.179041 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.179047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.180440 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.180454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.181329 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.181334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.181529 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.182061 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:05.182078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.185423 (+ 3345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260925 01:24:05.185998 16056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:24:05.186693 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:05.176655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:52915 (local address 127.6.236.190:40991)
0925 01:24:05.176792 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.176796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.176812 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:05.176861 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.176866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.176938 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.177018 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.177798 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.178856 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.180600 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.181185 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.181293 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.186458 (+ 5165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:52915: BlockingRecv error: recv got EOF from 127.6.236.129:52915 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:24:05.188249 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:24:05.197994 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16193
I20260925 01:24:05.216966 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.198955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.6.236.190:40991)
0925 01:24:05.199124 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.199130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.199739 (+ 609us) server_negotiation.cc:407] Connection header received
0925 01:24:05.199960 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.199966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.200038 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.200100 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.201965 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.203051 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.204890 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.205522 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.205601 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.208859 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:05.208870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:05.208891 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:05.208938 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:05.213850 (+ 4912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:05.214597 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:05.214603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:05.214610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:05.214776 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:05.215131 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:05.215136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:05.215139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:05.215625 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:05.215909 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:05.216337 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 01:24:05.216706 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:24:05.525746 16192 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.018s user 0.002s sys 0.009s
W20260925 01:24:05.527391 16191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:04.489123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:51707 (local address 127.6.236.190:40991)
0925 01:24:04.502344 (+ 13221us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:04.502350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:04.502368 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:24:04.502419 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:04.502423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:04.502486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:04.502583 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:04.504123 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.504845 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.506384 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.506793 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.506909 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.527002 (+1020093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:51707: BlockingRecv error: recv got EOF from 127.6.236.130:51707 (error 108)
Metrics: {"server-negotiator.queue_time_us":13082,"thread_start_us":12975,"threads_started":1}
I20260925 01:24:05.527740 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:04.489609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:51707)
0925 01:24:04.499533 (+ 9924us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:04.499573 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:04.501165 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:04.502852 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:04.502860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:04.503379 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:04.503953 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:04.503970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.505010 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.505015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:04.506226 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:04.506267 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:04.507024 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:04.507033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:04.507267 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:04.507876 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:04.507903 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.526688 (+1018785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9627,"thread_start_us":11757,"threads_started":1}
W20260925 01:24:05.528693 16189 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:40991 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.236.190:40991: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:24:05.538383 16191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:05.529442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:47687 (local address 127.6.236.190:40991)
0925 01:24:05.529609 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.529615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.529637 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:24:05.529781 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.529786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.529861 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.529937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.530715 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.531518 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.532797 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.533273 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.533372 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.538122 (+ 4750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:47687: BlockingRecv error: recv got EOF from 127.6.236.130:47687 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:24:05.539148 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.529295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:47687)
0925 01:24:05.529476 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:05.529496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:05.529645 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:05.529982 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:05.529986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:05.530394 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:05.530583 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.530593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.531660 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.531666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.532669 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.532683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.533440 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.533447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.533621 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.534170 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:05.534188 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.538891 (+ 4703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:05 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:05.548856 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.539813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:34451)
0925 01:24:05.540078 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:05.540098 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:05.540211 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:05.540496 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:05.540501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:05.540925 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:05.541141 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.541153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.542118 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.542123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.543795 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.543812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.544461 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.544467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.544669 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.545229 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:05.545253 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:05.548685 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154}
W20260925 01:24:05.549202 16189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:24:05.549239 16191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:05.539929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:34451 (local address 127.6.236.190:40991)
0925 01:24:05.540084 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.540090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.540144 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:24:05.540305 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.540308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.540370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.540441 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.541264 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.542001 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.543939 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.544330 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.544437 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.548953 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:34451: BlockingRecv error: recv got EOF from 127.6.236.130:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:06.200754 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:06.186614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:58717)
0925 01:24:06.192019 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:06.192043 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:06.192187 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:06.192705 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:06.192710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:06.193253 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:06.193442 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.193455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.194820 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.194823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.195753 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.195764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.196671 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.196675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.196800 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.197381 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:06.197398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:06.200504 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5260,"thread_start_us":5114,"threads_started":1}
W20260925 01:24:06.201018 16329 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:06.186658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58717 (local address 127.6.236.190:40991)
0925 01:24:06.192284 (+ 5626us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:06.192290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:06.192311 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:24:06.192371 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:06.192376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:06.192448 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:06.192563 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:06.193569 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.194703 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.195866 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.196507 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.196617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.200747 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:58717: BlockingRecv error: recv got EOF from 127.6.236.129:58717 (error 108)
Metrics: {"server-negotiator.queue_time_us":5510,"thread_start_us":5396,"threads_started":1}
W20260925 01:24:06.209694 16325 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.979s user 0.002s sys 0.009s
W20260925 01:24:06.210745 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:05.209569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:59083 (local address 127.6.236.190:40991)
0925 01:24:05.218127 (+ 8558us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:05.218132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:05.223176 (+ 5044us) server_negotiation.cc:407] Connection header received
0925 01:24:05.225141 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:05.225147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:05.225239 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:05.225317 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:05.227079 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.227789 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.229302 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.229703 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.229806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.210526 (+980720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:59083: BlockingRecv error: recv got EOF from 127.6.236.131:59083 (error 108)
Metrics: {"server-negotiator.queue_time_us":8455,"thread_start_us":12871,"threads_started":1}
I20260925 01:24:06.211282 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:05.208937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:59083)
0925 01:24:05.223033 (+ 14096us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:05.223074 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:05.224899 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:05.225554 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:05.225562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:05.226092 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:05.226920 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.226946 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.227935 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.227939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.229173 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:05.229185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:05.229906 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:05.229914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:05.230128 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:05.231010 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:05.231039 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:06.210389 (+979350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13807,"thread_start_us":13584,"threads_started":1}
W20260925 01:24:06.212072 16322 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:40991 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.236.190:40991: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:06.220290 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:06.212605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:50295)
0925 01:24:06.212769 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:06.212798 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:06.212890 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:06.213182 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:06.213186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:06.213505 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:06.213673 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.213681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.214620 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.214624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.215587 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.215599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.216324 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.216328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.216444 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.217023 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:06.217045 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:06.220095 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 01:24:06.220420 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:06.212679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50295 (local address 127.6.236.190:40991)
0925 01:24:06.212907 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:06.212911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:06.212944 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:24:06.212985 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:06.212988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:06.213042 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:06.213110 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:06.213781 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.214496 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.215714 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.216185 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.216305 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.220254 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:50295: BlockingRecv error: recv got EOF from 127.6.236.131:50295 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:06.227763 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:06.220857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:44537)
0925 01:24:06.220992 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:06.221007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:06.221085 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:06.221334 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:06.221337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:06.221621 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:06.221756 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.221763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.222699 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.222702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.223600 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.223610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.224216 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.224220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.224347 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.224804 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:06.224819 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:06.227577 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 01:24:06.228020 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:06.220925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:44537 (local address 127.6.236.190:40991)
0925 01:24:06.221066 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:06.221071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:06.221109 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:24:06.221182 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:06.221187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:06.221242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:06.221311 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:06.221858 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.222601 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.223698 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.224097 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.224180 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.227774 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:44537: BlockingRecv error: recv got EOF from 127.6.236.131:44537 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 01:24:06.228021 16322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:06 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:06.558120 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:06.549753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:36511)
0925 01:24:06.550232 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:06.550267 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:06.550380 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:06.550640 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:06.550643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:06.551046 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:06.551210 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.551219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.552115 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.552118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.553080 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:06.553092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.553758 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.553763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:06.553919 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.554569 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:06.554591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:06.557869 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
W20260925 01:24:06.558306 16058 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:06.549814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:36511 (local address 127.6.236.190:40991)
0925 01:24:06.550046 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:06.550052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:06.550336 (+ 284us) server_negotiation.cc:407] Connection header received
0925 01:24:06.550467 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:06.550470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:06.550532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:06.550608 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:06.551316 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.552007 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.553205 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:06.553608 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:06.553727 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:06.558075 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:36511: BlockingRecv error: recv got EOF from 127.6.236.130:36511 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:07.212195 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:07.201571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:55167)
0925 01:24:07.202386 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:07.202406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:07.202545 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:07.202954 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:07.202959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:07.203485 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:07.203740 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.203755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.204912 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.204918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.205945 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.205959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.206827 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.206834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.207017 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.207669 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:07.207692 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:07.211864 (+ 4172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":557,"threads_started":1}
W20260925 01:24:07.212421 16332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:07.201690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55167 (local address 127.6.236.190:40991)
0925 01:24:07.202199 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:07.202205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:07.202479 (+ 274us) server_negotiation.cc:407] Connection header received
0925 01:24:07.202706 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:07.202711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:07.202815 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:07.202926 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:07.203924 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.204777 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.206067 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.206669 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.206762 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.212081 (+ 5319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:55167: BlockingRecv error: recv got EOF from 127.6.236.129:55167 (error 108)
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:07.237165 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:07.228523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:60141)
0925 01:24:07.229184 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:07.229199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:07.229306 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:07.229585 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:07.229588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:07.229979 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:07.230122 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.230130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.231109 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.231113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.232074 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.232087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.232755 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.232761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.232908 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.233419 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:07.233438 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:07.236931 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":244,"threads_started":1}
W20260925 01:24:07.237438 16332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:07.228599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:60141 (local address 127.6.236.190:40991)
0925 01:24:07.228803 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:07.228809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:07.229266 (+ 457us) server_negotiation.cc:407] Connection header received
0925 01:24:07.229396 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:07.229400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:07.229462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:07.229557 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:07.230225 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.230981 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.232186 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.232614 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.232704 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.237125 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:60141: BlockingRecv error: recv got EOF from 127.6.236.131:60141 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:07 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:07.567678 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:07.559015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:40647)
0925 01:24:07.559492 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:07.559507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:07.559634 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:07.559934 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:07.559939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:07.560357 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:07.560533 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.560542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.561540 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.561546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.562594 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:07.562607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.563271 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.563277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:07.563434 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.563941 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:07.563958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:07.567414 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
W20260925 01:24:07.567793 16332 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:07.559120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:40647 (local address 127.6.236.190:40991)
0925 01:24:07.559308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:07.559315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:07.559604 (+ 289us) server_negotiation.cc:407] Connection header received
0925 01:24:07.559740 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:07.559745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:07.559808 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:07.559883 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:07.560672 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.561395 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.562728 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:07.563126 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:07.563231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:07.567579 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:40647: BlockingRecv error: recv got EOF from 127.6.236.130:40647 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:08.221014 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:08.213103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:43049)
0925 01:24:08.213630 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:08.213645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:08.213745 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:08.214008 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:08.214011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:08.214403 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:08.214599 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.214609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.215550 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.215554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.216441 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.216461 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.217017 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.217021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.217133 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.217656 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:08.217673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:08.220708 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":218,"threads_started":1}
W20260925 01:24:08.221163 16336 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:08.213258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:43049 (local address 127.6.236.190:40991)
0925 01:24:08.213716 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:08.213720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:08.213736 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:08.213837 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:08.213841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:08.213900 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:08.213980 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:08.214706 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.215436 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.216549 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.216922 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.217004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.220909 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:43049: BlockingRecv error: recv got EOF from 127.6.236.129:43049 (error 108)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:08.246411 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:08.238055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:48531)
0925 01:24:08.238496 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:08.238510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:08.238636 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:08.238932 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:08.238936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:08.239351 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:08.239517 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.239526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.240519 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.240522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.241646 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.241662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.242436 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.242442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.242590 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.243138 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:08.243157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:08.246136 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
W20260925 01:24:08.246541 16336 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:08.238104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:48531 (local address 127.6.236.190:40991)
0925 01:24:08.238289 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:08.238295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:08.238583 (+ 288us) server_negotiation.cc:407] Connection header received
0925 01:24:08.238748 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:08.238754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:08.238814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:08.238891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:08.239644 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.240380 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.241773 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.242289 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.242407 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.246341 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:48531: BlockingRecv error: recv got EOF from 127.6.236.131:48531 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:08 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:08.577320 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:08.568726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:59425)
0925 01:24:08.569289 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:08.569305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:08.569433 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:08.569792 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:08.569797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:08.570176 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:08.570370 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.570380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.571376 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.571381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.572274 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:08.572285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.573007 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.573013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:08.573184 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.573760 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:08.573782 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:08.577093 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":296,"threads_started":1}
W20260925 01:24:08.577572 16336 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:08.568841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:59425 (local address 127.6.236.190:40991)
0925 01:24:08.569033 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:08.569040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:08.569389 (+ 349us) server_negotiation.cc:407] Connection header received
0925 01:24:08.569554 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:08.569558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:08.569641 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:08.569726 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:08.570516 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.571232 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.572407 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:08.572851 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:08.572966 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:08.577331 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:59425: BlockingRecv error: recv got EOF from 127.6.236.130:59425 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:09.230964 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:09.221946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:47463)
0925 01:24:09.222761 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:09.222782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:09.222924 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:09.223296 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:09.223301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:09.223757 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:09.223923 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.223932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.224958 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.224964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.225938 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.225950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.226646 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.226651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.226827 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.227362 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:09.227380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:09.230628 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":554,"threads_started":1}
W20260925 01:24:09.231112 16340 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:09.222017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:47463 (local address 127.6.236.190:40991)
0925 01:24:09.222560 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:09.222566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:09.222861 (+ 295us) server_negotiation.cc:407] Connection header received
0925 01:24:09.223048 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:09.223053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:09.223143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:09.223228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:09.224077 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.224831 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.226080 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.226511 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.226619 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.230837 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:47463: BlockingRecv error: recv got EOF from 127.6.236.129:47463 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":288,"threads_started":1}
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:09.255115 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:09.247239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:57601)
0925 01:24:09.247727 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:09.247741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:09.247851 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:09.248099 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:09.248102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:09.248441 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:09.248599 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.248609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.249497 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.249500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.250451 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.250465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.251034 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.251038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.251149 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.251702 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:09.251721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:09.254877 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":218,"threads_started":1}
W20260925 01:24:09.255270 16340 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:09.247269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57601 (local address 127.6.236.190:40991)
0925 01:24:09.247452 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:09.247457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:09.247799 (+ 342us) server_negotiation.cc:407] Connection header received
0925 01:24:09.247936 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:09.247939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:09.247996 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:09.248078 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:09.248706 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.249396 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.250557 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.250931 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.251018 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.255048 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:57601: BlockingRecv error: recv got EOF from 127.6.236.131:57601 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:09 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:09.597908 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:09.578216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:35875)
0925 01:24:09.578704 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:09.578719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:09.578848 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:09.579181 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:09.579185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:09.579558 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:09.579716 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.579725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.580728 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.580733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.581733 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:09.581746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.582460 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.582466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:09.582647 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.593943 (+ 11296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:09.593964 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:09.597572 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
W20260925 01:24:09.598086 16340 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:09.578282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:35875 (local address 127.6.236.190:40991)
0925 01:24:09.578450 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:09.578456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:09.578798 (+ 342us) server_negotiation.cc:407] Connection header received
0925 01:24:09.578974 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:09.578977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:09.579044 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:09.579126 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:09.579850 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.580604 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.581864 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:09.582309 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:09.582420 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:09.597831 (+ 15411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:35875: BlockingRecv error: recv got EOF from 127.6.236.130:35875 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:10.240734 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:10.231920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:38721)
0925 01:24:10.232451 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:10.232468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:10.232585 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:10.232894 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:10.232897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:10.233384 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:10.233566 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.233576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.234619 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.234623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.235866 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.235881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.236656 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.236663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.236844 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.237364 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:10.237380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:10.240454 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":254,"threads_started":1}
W20260925 01:24:10.240909 16344 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:10.232046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:38721 (local address 127.6.236.190:40991)
0925 01:24:10.232514 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:10.232518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:10.232534 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:10.232677 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:10.232681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:10.232752 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:10.232837 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:10.233698 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.234486 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.235996 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.236483 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.236600 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.240689 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:38721: BlockingRecv error: recv got EOF from 127.6.236.129:38721 (error 108)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":244,"threads_started":1}
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:10.264822 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:10.256028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:58605)
0925 01:24:10.256556 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:10.256572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:10.256681 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:10.257002 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:10.257005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:10.257365 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:10.257523 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.257530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.258552 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.258556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.259964 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.259981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.260761 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.260766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.260887 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.261391 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:10.261407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:10.264580 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
W20260925 01:24:10.265093 16344 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:10.256060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:58605 (local address 127.6.236.190:40991)
0925 01:24:10.256243 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:10.256247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:10.256653 (+ 406us) server_negotiation.cc:407] Connection header received
0925 01:24:10.256778 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:10.256782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:10.256845 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:10.256926 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:10.257660 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.258417 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.260138 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.260639 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.260737 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.264841 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:58605: BlockingRecv error: recv got EOF from 127.6.236.131:58605 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:10 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:10.607084 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:10.598969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:54915)
0925 01:24:10.599451 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:10.599467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:10.599593 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:10.599960 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:10.599966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:10.600367 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:10.600512 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.600521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.601487 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.601492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.602454 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:10.602468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.603144 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.603151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:10.603308 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.603820 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:10.603838 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:10.606867 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
W20260925 01:24:10.607286 16344 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:10.599144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:54915 (local address 127.6.236.190:40991)
0925 01:24:10.599370 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:10.599377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:10.599545 (+ 168us) server_negotiation.cc:407] Connection header received
0925 01:24:10.599702 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:10.599708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:10.599810 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:10.599901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:10.600644 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.601347 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.602581 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:10.602996 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:10.603102 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:10.607045 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:54915: BlockingRecv error: recv got EOF from 127.6.236.130:54915 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:11.249900 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:11.241600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:51385)
0925 01:24:11.242083 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:11.242100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:11.242211 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:11.242507 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:11.242510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:11.242892 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:11.243125 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.243138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.244079 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.244083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.245028 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.245040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.245677 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.245681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.245811 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.246381 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:11.246399 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:11.249593 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":236,"threads_started":1}
W20260925 01:24:11.250051 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:11.241651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:51385 (local address 127.6.236.190:40991)
0925 01:24:11.242096 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:11.242101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:11.242152 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:24:11.242323 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:11.242327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:11.242392 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:11.242475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:11.243239 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.243957 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.245143 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.245546 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.245638 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.249810 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:51385: BlockingRecv error: recv got EOF from 127.6.236.129:51385 (error 108)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:11.273564 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:11.265702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:57575)
0925 01:24:11.266128 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:11.266144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:11.266326 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:11.266630 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:11.266634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:11.266987 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:11.267126 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.267133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.268106 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.268109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.269061 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.269074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.269742 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.269746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.269856 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.270386 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:11.270403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:11.273340 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":216,"threads_started":1}
W20260925 01:24:11.273741 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:11.265780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57575 (local address 127.6.236.190:40991)
0925 01:24:11.265975 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:11.265981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:11.266230 (+ 249us) server_negotiation.cc:407] Connection header received
0925 01:24:11.266439 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:11.266442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:11.266512 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:11.266587 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:11.267262 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.268007 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.269187 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.269631 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.269720 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.273520 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:57575: BlockingRecv error: recv got EOF from 127.6.236.131:57575 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:11 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:11.616258 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:11.608105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:39617)
0925 01:24:11.608536 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:11.608553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:11.608664 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:11.608954 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:11.608957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:11.609340 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:11.609503 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.609511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.610503 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.610506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.611404 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:11.611416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.611997 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.612001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:11.612121 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.612640 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:11.612657 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:11.615949 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
W20260925 01:24:11.616400 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:11.608162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:39617 (local address 127.6.236.190:40991)
0925 01:24:11.608384 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:11.608390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:11.608625 (+ 235us) server_negotiation.cc:407] Connection header received
0925 01:24:11.608775 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:11.608778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:11.608843 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:11.608923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:11.609640 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.610390 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.611506 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:11.611880 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:11.611974 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:11.616162 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:39617: BlockingRecv error: recv got EOF from 127.6.236.130:39617 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:12.259582 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:12.250877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:40789)
0925 01:24:12.251439 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:12.251459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:12.251594 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:12.251965 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:12.251970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:12.252402 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:12.252612 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.252624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.253764 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.253768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.254754 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.254767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.255463 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.255469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.255618 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.256146 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:12.256162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:12.259340 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":244,"threads_started":1}
W20260925 01:24:12.259744 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:12.250942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:40789 (local address 127.6.236.190:40991)
0925 01:24:12.251699 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:12.251705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:12.251721 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:12.251763 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:12.251767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:12.251821 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:12.251906 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:12.252751 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.253633 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.254883 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.255326 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.255431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.259516 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:40789: BlockingRecv error: recv got EOF from 127.6.236.129:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":512,"threads_started":1}
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:12.282820 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:12.274388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:55777)
0925 01:24:12.274835 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:12.274851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:12.274959 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:12.275236 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:12.275240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:12.275589 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:12.275809 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.275824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.277064 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.277068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.277975 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.277988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.278683 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.278687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.278805 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.279284 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:12.279300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:12.282516 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
W20260925 01:24:12.282932 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:12.274450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:55777 (local address 127.6.236.190:40991)
0925 01:24:12.274603 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:12.274608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:12.274938 (+ 330us) server_negotiation.cc:407] Connection header received
0925 01:24:12.275055 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:12.275058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:12.275128 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:12.275208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:12.275956 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.276937 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.278101 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.278555 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.278651 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.282724 (+ 4073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:55777: BlockingRecv error: recv got EOF from 127.6.236.131:55777 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:12 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:12.625552 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:12.617195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:48507)
0925 01:24:12.617768 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:12.617783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:12.617909 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:12.618205 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:12.618210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:12.618644 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:12.618819 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.618828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.619819 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.619823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.620754 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:12.620766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.621455 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.621461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:12.621632 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.622145 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:12.622162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:12.625223 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":262,"threads_started":1}
W20260925 01:24:12.625864 16351 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:12.617274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:48507 (local address 127.6.236.190:40991)
0925 01:24:12.617499 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:12.617507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:12.617867 (+ 360us) server_negotiation.cc:407] Connection header received
0925 01:24:12.618016 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:12.618020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:12.618082 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:12.618160 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:12.618954 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.619672 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.620887 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:12.621314 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:12.621421 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:12.625443 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:48507: BlockingRecv error: recv got EOF from 127.6.236.130:48507 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:13.268805 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:13.260462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:48143)
0925 01:24:13.260980 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:13.260996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:13.261103 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:13.261368 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:13.261371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:13.261793 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:13.261969 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.261978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.262972 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.262976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.263904 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.263915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.264574 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.264579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.264706 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.265237 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:13.265254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:13.268474 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
W20260925 01:24:13.268926 16356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:13.260541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:48143 (local address 127.6.236.190:40991)
0925 01:24:13.261118 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:13.261123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:13.261139 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:13.261183 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:13.261187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:13.261252 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:13.261337 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:13.262094 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.262859 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.264010 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.264436 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.264521 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.268701 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:48143: BlockingRecv error: recv got EOF from 127.6.236.129:48143 (error 108)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":314,"threads_started":1}
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:13.292439 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:13.283649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:35791)
0925 01:24:13.284193 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:13.284209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:13.284318 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:13.284569 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:13.284572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:13.284942 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:13.285115 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.285124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.286087 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.286092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.287247 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.287264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.288090 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.288095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.288222 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.288806 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:13.288830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:13.292105 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":245,"threads_started":1}
W20260925 01:24:13.292482 16356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:13.283774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:35791 (local address 127.6.236.190:40991)
0925 01:24:13.283924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:13.283929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:13.284268 (+ 339us) server_negotiation.cc:407] Connection header received
0925 01:24:13.284407 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:13.284411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:13.284467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:13.284541 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:13.285245 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.285966 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.287388 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.287966 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.288061 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.292275 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:35791: BlockingRecv error: recv got EOF from 127.6.236.131:35791 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:13 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:13.635147 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:13.626402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:39869)
0925 01:24:13.626861 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:13.626877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:13.626993 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:13.627255 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:13.627257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:13.627670 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:13.627862 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.627877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.629310 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.629314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.630344 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:13.630358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.631012 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.631017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:13.631132 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.631665 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:13.631682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:13.634907 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
W20260925 01:24:13.635349 16356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:13.626506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:39869 (local address 127.6.236.190:40991)
0925 01:24:13.626747 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:13.626753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:13.626951 (+ 198us) server_negotiation.cc:407] Connection header received
0925 01:24:13.627084 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:13.627087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:13.627149 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:13.627225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:13.628020 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.629149 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.630471 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:13.630854 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:13.630939 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:13.635096 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:39869: BlockingRecv error: recv got EOF from 127.6.236.130:39869 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:14.278319 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:14.269838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:58961)
0925 01:24:14.270382 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:14.270404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:14.270529 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:14.270883 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:14.270889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:14.271292 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:14.271456 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.271469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.272547 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.272552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.273469 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.273481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.274183 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.274188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.274353 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.274857 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:14.274873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:14.278044 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":234,"threads_started":1}
W20260925 01:24:14.278549 16360 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:14.269893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58961 (local address 127.6.236.190:40991)
0925 01:24:14.270592 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:14.270597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:14.270613 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:14.270661 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:14.270666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:14.270743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:14.270828 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:14.271624 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.272412 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.273590 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.274039 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.274150 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.278234 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:58961: BlockingRecv error: recv got EOF from 127.6.236.129:58961 (error 108)
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":481,"threads_started":1}
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:14.301266 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:14.293293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:47179)
0925 01:24:14.293769 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:14.293785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:14.293893 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:14.294171 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:14.294175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:14.294645 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:14.294816 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.294825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.295766 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.295770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.296755 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.296767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.297448 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.297452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.297566 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.298081 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:14.298096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:14.301007 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
W20260925 01:24:14.301368 16360 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:14.293344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:47179 (local address 127.6.236.190:40991)
0925 01:24:14.293534 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:14.293540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:14.293850 (+ 310us) server_negotiation.cc:407] Connection header received
0925 01:24:14.293995 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:14.293999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:14.294058 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:14.294142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:14.294935 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.295662 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.296878 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.297310 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.297401 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.301187 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:47179: BlockingRecv error: recv got EOF from 127.6.236.131:47179 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:14 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:14.647904 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:14.636026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:36521)
0925 01:24:14.636504 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:14.636520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:14.636629 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:14.636940 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:14.636943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:14.637330 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:14.637472 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.637480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.638438 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.638442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.639393 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:14.639409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.640192 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.640197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:14.640366 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.644434 (+ 4068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:14.644452 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:14.647682 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
W20260925 01:24:14.648120 16360 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:14.636086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:36521 (local address 127.6.236.190:40991)
0925 01:24:14.636244 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:14.636248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:14.636585 (+ 337us) server_negotiation.cc:407] Connection header received
0925 01:24:14.636723 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:14.636726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:14.636828 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:14.636912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:14.637584 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.638326 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.639508 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:14.640026 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:14.640146 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:14.647861 (+ 7715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:36521: BlockingRecv error: recv got EOF from 127.6.236.130:36521 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:15.287601 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:15.279306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:44107)
0925 01:24:15.279798 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:15.279814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:15.279917 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:15.280333 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:15.280336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:15.280689 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:15.280846 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.280854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.281814 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.281817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.282820 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.282832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.283463 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.283466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.283573 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.284122 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:15.284139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:15.287358 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
W20260925 01:24:15.287770 16364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:15.279402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:44107 (local address 127.6.236.190:40991)
0925 01:24:15.280090 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:15.280094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:15.280109 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:24:15.280153 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:15.280156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:15.280216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:15.280302 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:15.280968 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.281712 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.282932 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.283351 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.283442 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.287538 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:44107: BlockingRecv error: recv got EOF from 127.6.236.129:44107 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":487,"threads_started":1}
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:15.310032 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:15.302056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:33713)
0925 01:24:15.302551 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:15.302567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:15.302671 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:15.302930 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:15.302934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:15.303316 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:15.303453 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.303461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.304390 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.304393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.305305 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.305316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.305890 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.305894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.306007 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.306541 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:15.306558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:15.309822 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
W20260925 01:24:15.310195 16364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:15.302131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:33713 (local address 127.6.236.190:40991)
0925 01:24:15.302306 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:15.302310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:15.302627 (+ 317us) server_negotiation.cc:407] Connection header received
0925 01:24:15.302763 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:15.302766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:15.302826 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:15.302904 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:15.303554 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.304288 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.305406 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.305786 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.305868 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.309986 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:33713: BlockingRecv error: recv got EOF from 127.6.236.131:33713 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:15 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:15.656642 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:15.648793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:52301)
0925 01:24:15.649217 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:15.649233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:15.649340 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:15.649613 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:15.649616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:15.649983 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:15.650140 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.650148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.651069 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.651073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.652055 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:15.652066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.652717 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.652721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:15.652829 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.653362 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:15.653379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:15.656407 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
W20260925 01:24:15.656807 16364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:15.648854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:52301 (local address 127.6.236.190:40991)
0925 01:24:15.649044 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:15.649050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:15.649300 (+ 250us) server_negotiation.cc:407] Connection header received
0925 01:24:15.649428 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:15.649432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:15.649506 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:15.649590 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:15.650266 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.650957 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.652181 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:15.652589 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:15.652690 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:15.656597 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:52301: BlockingRecv error: recv got EOF from 127.6.236.130:52301 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:16.296836 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:16.288416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:33163)
0925 01:24:16.288920 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:16.288939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:16.289042 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:16.289335 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:16.289338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:16.289866 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:16.290095 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.290107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.291344 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.291347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.292288 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.292300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.292917 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.292921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.293040 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.293547 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:16.293564 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:16.296617 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
W20260925 01:24:16.297061 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:16.288589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:33163 (local address 127.6.236.190:40991)
0925 01:24:16.289031 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:16.289035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:16.289051 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:16.289137 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:16.289140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:16.289203 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:16.289319 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:16.290233 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.291232 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.292388 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.292809 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.292894 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.296811 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:33163: BlockingRecv error: recv got EOF from 127.6.236.129:33163 (error 108)
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:16.318993 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:16.310915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:55029)
0925 01:24:16.311374 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:16.311389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:16.311495 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:16.311782 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:16.311785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:16.312139 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:16.312304 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.312312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.313361 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.313367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.314509 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.314522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.315134 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.315137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.315250 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.315746 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:16.315762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:16.318762 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":264,"threads_started":1}
W20260925 01:24:16.319105 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:16.310953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:55029 (local address 127.6.236.190:40991)
0925 01:24:16.311127 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:16.311134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:16.311457 (+ 323us) server_negotiation.cc:407] Connection header received
0925 01:24:16.311617 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:16.311622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:16.311685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:16.311755 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:16.312406 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.313227 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.314624 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.315026 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.315124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.318921 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:55029: BlockingRecv error: recv got EOF from 127.6.236.131:55029 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:16 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:16.665836 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:16.657511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:38633)
0925 01:24:16.657953 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:16.657969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:16.658081 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:16.658378 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:16.658381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:16.658771 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:16.658941 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.658948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.659920 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.659924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.660821 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:16.660833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.661438 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.661443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:16.661573 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.662128 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:16.662153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:16.665620 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":246,"threads_started":1}
W20260925 01:24:16.666062 16368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:16.657583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:38633 (local address 127.6.236.190:40991)
0925 01:24:16.657780 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:16.657787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:16.658043 (+ 256us) server_negotiation.cc:407] Connection header received
0925 01:24:16.658183 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:16.658187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:16.658270 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:16.658348 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:16.659040 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.659822 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.660924 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:16.661323 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:16.661408 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:16.665799 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:38633: BlockingRecv error: recv got EOF from 127.6.236.130:38633 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:17.306056 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:17.297673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:39249)
0925 01:24:17.298228 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:17.298273 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:17.298385 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:17.298654 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:17.298657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:17.299055 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:17.299238 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.299248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.300183 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.300186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.301114 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.301125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.301778 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.301782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.301901 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.302464 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:17.302482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:17.305820 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":251,"threads_started":1}
W20260925 01:24:17.306202 16372 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:17.297751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:39249 (local address 127.6.236.190:40991)
0925 01:24:17.298306 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:17.298311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:17.298328 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:24:17.298474 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:17.298478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:17.298545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:17.298627 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:17.299359 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.300069 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.301219 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.301647 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.301739 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.306006 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:39249: BlockingRecv error: recv got EOF from 127.6.236.129:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:17.328236 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:17.319837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:56267)
0925 01:24:17.320259 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:17.320274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:17.320379 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:17.320732 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:17.320736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:17.321104 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:17.321292 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.321301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.322306 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.322310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.323189 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.323200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.323841 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.323845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.323967 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.324487 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:17.324504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:17.327947 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":222,"threads_started":1}
W20260925 01:24:17.328610 16372 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:17.319915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:56267 (local address 127.6.236.190:40991)
0925 01:24:17.320105 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:17.320110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:17.320340 (+ 230us) server_negotiation.cc:407] Connection header received
0925 01:24:17.320480 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:17.320483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:17.320587 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:17.320692 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:17.321416 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.322182 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.323313 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.323726 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.323814 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.328147 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:56267: BlockingRecv error: recv got EOF from 127.6.236.131:56267 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:17 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:17.675474 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:17.666762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:44259)
0925 01:24:17.667268 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:17.667284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:17.667408 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:17.667742 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:17.667748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:17.668271 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:17.668492 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.668505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.669529 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.669535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.670486 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:17.670499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.671213 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.671220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:17.671409 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.671952 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:17.671970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:17.675239 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":278,"threads_started":1}
W20260925 01:24:17.675747 16372 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:17.666837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:44259 (local address 127.6.236.190:40991)
0925 01:24:17.667017 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:17.667024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:17.667363 (+ 339us) server_negotiation.cc:407] Connection header received
0925 01:24:17.667518 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:17.667523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:17.667597 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:17.667677 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:17.668624 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.669393 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.670629 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:17.671047 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:17.671155 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:17.675420 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:44259: BlockingRecv error: recv got EOF from 127.6.236.130:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:18.315076 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:18.306877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:47265)
0925 01:24:18.307327 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:18.307343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:18.307442 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:18.307728 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:18.307730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:18.308106 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:18.308244 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.308252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.309252 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.309256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.310150 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.310162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.310850 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.310854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.310981 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.311532 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:18.311549 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:18.314776 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
W20260925 01:24:18.315253 16376 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:18.306917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:47265 (local address 127.6.236.190:40991)
0925 01:24:18.307398 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:18.307402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:18.307418 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:24:18.307561 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:18.307565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:18.307626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:18.307714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:18.308362 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.309137 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.310310 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.310736 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.310827 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.314975 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:47265: BlockingRecv error: recv got EOF from 127.6.236.129:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":293,"threads_started":1}
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:18.337054 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:18.329042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:42247)
0925 01:24:18.329539 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:18.329554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:18.329667 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:18.329922 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:18.329925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:18.330347 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:18.330517 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.330526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.331404 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.331407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.332367 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.332379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.333036 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.333040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.333214 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.333784 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:18.333802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:18.336822 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
W20260925 01:24:18.337165 16376 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:18.329083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:42247 (local address 127.6.236.190:40991)
0925 01:24:18.329235 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:18.329239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:18.329617 (+ 378us) server_negotiation.cc:407] Connection header received
0925 01:24:18.329767 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:18.329771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:18.329828 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:18.329894 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:18.330619 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.331293 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.332470 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.332881 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.332970 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.336996 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:42247: BlockingRecv error: recv got EOF from 127.6.236.131:42247 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:18 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:18.684895 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:18.676349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:50961)
0925 01:24:18.676896 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:18.676915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:18.677047 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:18.677426 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:18.677432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:18.677935 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:18.678126 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.678140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.679299 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.679303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.680198 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:18.680210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.680840 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.680844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:18.680960 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.681537 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:18.681555 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:18.684667 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":319,"threads_started":1}
W20260925 01:24:18.685119 16376 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:18.676499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:50961 (local address 127.6.236.190:40991)
0925 01:24:18.676666 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:18.676672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:18.676998 (+ 326us) server_negotiation.cc:407] Connection header received
0925 01:24:18.677193 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:18.677197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:18.677271 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:18.677369 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:18.678287 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.679172 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.680314 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:18.680735 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:18.680825 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:18.684863 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:50961: BlockingRecv error: recv got EOF from 127.6.236.130:50961 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.129@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:19.324869 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:19.316241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.129:59833)
0925 01:24:19.316905 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:19.316921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:19.317038 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:19.317400 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:19.317405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:19.317836 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:19.318061 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.318075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.319123 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.319126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.320047 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.320059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.320752 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.320756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.320883 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.321454 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:19.321472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:19.324627 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":424,"threads_started":1}
W20260925 01:24:19.325026 16380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:19.316341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:59833 (local address 127.6.236.190:40991)
0925 01:24:19.316792 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:19.316799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:19.317005 (+ 206us) server_negotiation.cc:407] Connection header received
0925 01:24:19.317157 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:19.317161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:19.317246 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:19.317345 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:19.318219 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.318993 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.320164 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.320629 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.320719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.324799 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.129:59833: BlockingRecv error: recv got EOF from 127.6.236.129:59833 (error 108)
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.131@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:19.346450 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:19.337999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.131:48293)
0925 01:24:19.338432 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:19.338447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:19.338551 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:19.338833 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:19.338836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:19.339189 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:19.339372 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.339383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.340397 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.340402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.341519 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.341532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.342171 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.342193 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.342367 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.342915 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:19.342933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:19.346108 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
W20260925 01:24:19.346575 16380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:19.338052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:48293 (local address 127.6.236.190:40991)
0925 01:24:19.338281 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:19.338285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:19.338505 (+ 220us) server_negotiation.cc:407] Connection header received
0925 01:24:19.338658 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:19.338661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:19.338727 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:19.338802 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:19.339478 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.340215 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.341648 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.342038 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.342143 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.346346 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.131:48293: BlockingRecv error: recv got EOF from 127.6.236.131:48293 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:24:19 dist-test-slave-s0b2 krb5kdc[15322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.236.130@KRBTEST.COM for oryx/127.6.236.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:24:19.698863 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:19.685950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:40991 (local address 127.6.236.130:37097)
0925 01:24:19.686478 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:19.686532 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:19.686670 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:19.687008 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:19.687012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:19.687419 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:19.687621 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.687630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.688678 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.688683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.689610 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:19.689623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.690322 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.690328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:19.690516 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.695195 (+ 4679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:19.695214 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:19.698525 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.236.190:40991: Server oryx/127.6.236.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
W20260925 01:24:19.698976 16380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:19.686041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:37097 (local address 127.6.236.190:40991)
0925 01:24:19.686296 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:19.686302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:19.686622 (+ 320us) server_negotiation.cc:407] Connection header received
0925 01:24:19.686788 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:19.686794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:19.686875 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:19.686956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:19.687764 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.688532 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.689755 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:19.690174 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:19.690311 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:19.698744 (+ 8433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.236.130:37097: BlockingRecv error: recv got EOF from 127.6.236.130:37097 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:24:20.201730 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15909
I20260925 01:24:20.213145 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:20.202792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.6.236.129:37497)
0925 01:24:20.203384 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:20.203390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:20.203424 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:24:20.203487 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:20.203494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:20.203726 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:20.203886 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:20.205175 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.206186 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.207355 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.207894 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.207990 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:20.210300 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:20.210308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:20.210323 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:20.210378 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:20.212654 (+ 2276us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:24:20.212941 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49612: Request ticket server kudu/127.6.236.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":289,"threads_started":1}
W20260925 01:24:20.213243 16384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49612: Request ticket server kudu/127.6.236.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:24:20.213294 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15909: Not authorized: Client connection negotiation failed: client connection to 127.6.236.129:37497: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:24:20.213511 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15909
I20260925 01:24:20.227486 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16060
I20260925 01:24:20.239315 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:20.228393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.6.236.130:43617)
0925 01:24:20.229093 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:20.229103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:20.229131 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:24:20.229217 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:20.229225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:20.229551 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:20.229761 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:20.230974 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.232002 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.233184 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.233799 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.233892 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:20.236172 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:20.236194 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:20.236208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:20.236251 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:20.238791 (+ 2540us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:24:20.239070 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44536: Request ticket server kudu/127.6.236.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":362,"threads_started":1}
W20260925 01:24:20.239418 16385 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44536: Request ticket server kudu/127.6.236.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:24:20.239491 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16060: Not authorized: Client connection negotiation failed: client connection to 127.6.236.130:43617: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:24:20.239663 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16060
I20260925 01:24:20.253731 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16193
I20260925 01:24:20.265532 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:20.254772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.6.236.131:34321)
0925 01:24:20.255350 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:20.255358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:20.255384 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:24:20.255452 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:20.255458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:20.255699 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:20.255865 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:20.257519 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.258606 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.259723 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.260254 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.260349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:20.262703 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:20.262711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:20.262725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:20.262763 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:20.265033 (+ 2270us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:24:20.265301 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44458: Request ticket server kudu/127.6.236.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
W20260925 01:24:20.265640 16386 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44458: Request ticket server kudu/127.6.236.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:24:20.265643 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16193: Not authorized: Client connection negotiation failed: client connection to 127.6.236.131:34321: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.236.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:24:20.265803 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16193
I20260925 01:24:20.279356 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 15840
I20260925 01:24:20.280596 15902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:20.396731 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 15840
2026-09-25T01:24:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20946 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T01:24:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:24:20Z Disabled control of system clock
I20260925 01:24:20.465301 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35033
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:24:20.778522 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:20.778815 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:20.778934 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:20.786051 16391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:24:20.786159 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:20.786253 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:20.786327 16391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:24:20.786383 16391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:24:20.795503 16391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:35033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:20.796765 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:20.798516 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:20.809731 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:20.809800 16397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:20.809764 16396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:20.810200 16391 server_base.cc:1034] running on GCE node
I20260925 01:24:20.810987 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:20.812282 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:20.813475 16391 hybrid_clock.cc:648] HybridClock initialized: now 1790299460813410 us; error 52 us; skew 500 ppm
I20260925 01:24:20.813809 16391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:20.815968 16391 webserver.cc:460] Webserver started at http://127.6.236.190:44045/ using document root <none> and password file <none>
I20260925 01:24:20.816630 16391 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:20.816732 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:20.817039 16391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:20.819845 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1fdbcf63eaf4411ea472a38bd9580534"
format_stamp: "Formatted at 2026-09-25 01:24:20 on dist-test-slave-s0b2"
server_key: "68fec5ecfc8f01c0395b62a8b004770c"
server_key_iv: "9820e43cefbbfb36a72e07426a8cec6f"
server_key_version: "encryptionkey@0"
I20260925 01:24:20.820570 16391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1fdbcf63eaf4411ea472a38bd9580534"
format_stamp: "Formatted at 2026-09-25 01:24:20 on dist-test-slave-s0b2"
server_key: "68fec5ecfc8f01c0395b62a8b004770c"
server_key_iv: "9820e43cefbbfb36a72e07426a8cec6f"
server_key_version: "encryptionkey@0"
I20260925 01:24:20.825384 16391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:24:20.828691 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:20.830017 16391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:20.830173 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1fdbcf63eaf4411ea472a38bd9580534"
format_stamp: "Formatted at 2026-09-25 01:24:20 on dist-test-slave-s0b2"
server_key: "68fec5ecfc8f01c0395b62a8b004770c"
server_key_iv: "9820e43cefbbfb36a72e07426a8cec6f"
server_key_version: "encryptionkey@0"
I20260925 01:24:20.830425 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:20.863706 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:20.864594 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:20.864877 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:20.881570 16391 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:35033
I20260925 01:24:20.881613 16456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:35033 every 8 connection(s)
I20260925 01:24:20.883170 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:24:20.886907 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:20.893559 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16391
I20260925 01:24:20.893761 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:24:20.894650 7090 external_mini_cluster.cc:1428] Setting key 42d4efc6d6a52bea137148829a2e5d26
I20260925 01:24:20.897928 16457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534: Bootstrap starting.
I20260925 01:24:20.901273 16457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:20.902347 16457 log.cc:824] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:20.905264 16457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534: No bootstrap required, opened a new log
I20260925 01:24:20.907848 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:20.896706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.6.236.190:35033)
0925 01:24:20.897675 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:20.897695 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:20.897737 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:24:20.899426 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:20.899449 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:20.899922 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:20.900225 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:20.900880 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.902183 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.903705 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:20.904359 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:20.904483 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:20.905641 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:20.905649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:20.905662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:20.906170 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:20.906281 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:20.906307 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:20.906403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:20.906610 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:24:20.906917 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":329,"threads_started":1}
I20260925 01:24:20.912942 16457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } }
I20260925 01:24:20.913278 16457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:20.913374 16457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fdbcf63eaf4411ea472a38bd9580534, State: Initialized, Role: FOLLOWER
I20260925 01:24:20.913935 16457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } }
I20260925 01:24:20.914139 16457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:24:20.914280 16457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:24:20.914429 16457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:20.916527 16457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } }
I20260925 01:24:20.916990 16457 leader_election.cc:302] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fdbcf63eaf4411ea472a38bd9580534; no voters:
I20260925 01:24:20.917498 16457 leader_election.cc:288] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:24:20.917614 16462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:24:20.918286 16462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [term 1 LEADER]: Becoming Leader. State: Replica: 1fdbcf63eaf4411ea472a38bd9580534, State: Running, Role: LEADER
I20260925 01:24:20.918773 16462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } }
I20260925 01:24:20.919325 16457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:24:20.921403 16463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fdbcf63eaf4411ea472a38bd9580534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } } }
I20260925 01:24:20.921811 16463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:20.922472 16464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fdbcf63eaf4411ea472a38bd9580534. Latest consensus state: current_term: 1 leader_uuid: "1fdbcf63eaf4411ea472a38bd9580534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fdbcf63eaf4411ea472a38bd9580534" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35033 } } }
I20260925 01:24:20.922874 16464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:20.923662 16470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:24:20.928718 16470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:24:20.937347 16470 catalog_manager.cc:1347] Generated new cluster ID: 256b5411d43c4bc1b18fcdf639537660
I20260925 01:24:20.937463 16470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:24:20.977787 16470 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:24:20.979112 16470 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:24:20.994494 16470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fdbcf63eaf4411ea472a38bd9580534: Generated new TSK 0
I20260925 01:24:20.995323 16470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:24:21.005704 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--builtin_ntp_servers=127.6.236.148:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 01:24:21.329437 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:21.329706 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:21.329795 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:21.336858 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:21.336944 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:21.337057 16481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:24:21.345840 16481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:21.347060 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:21.348816 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:21.361171 16486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:21.361246 16487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:21.361198 16489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:21.362289 16481 server_base.cc:1034] running on GCE node
I20260925 01:24:21.362944 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:21.363838 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:21.365039 16481 hybrid_clock.cc:648] HybridClock initialized: now 1790299461364981 us; error 49 us; skew 500 ppm
I20260925 01:24:21.365340 16481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:21.367583 16481 webserver.cc:460] Webserver started at http://127.6.236.129:41181/ using document root <none> and password file <none>
I20260925 01:24:21.368330 16481 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:21.368419 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:21.368721 16481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:21.371502 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "76bac2a814134a09986a3933c7d549c2"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "134656d430c52f5175107214dbde90f3"
server_key_iv: "c76b2a96ce7d8eb38db5978a4592d179"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.372246 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "76bac2a814134a09986a3933c7d549c2"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "134656d430c52f5175107214dbde90f3"
server_key_iv: "c76b2a96ce7d8eb38db5978a4592d179"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.377203 16481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:24:21.380764 16495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.382151 16481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:21.382325 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "76bac2a814134a09986a3933c7d549c2"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "134656d430c52f5175107214dbde90f3"
server_key_iv: "c76b2a96ce7d8eb38db5978a4592d179"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.382513 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:21.402966 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:21.403734 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:21.403977 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:21.404824 16481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:21.406291 16481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:21.406386 16481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.406463 16481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:21.406514 16481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.435395 16481 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:40979
I20260925 01:24:21.435463 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:40979 every 8 connection(s)
I20260925 01:24:21.436656 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:24:21.439445 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16481
I20260925 01:24:21.439635 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:24:21.440323 7090 external_mini_cluster.cc:1428] Setting key 396c7cfe1aef057b5f3a583ef1f4bad9
I20260925 01:24:21.443254 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--builtin_ntp_servers=127.6.236.148:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:24:21.449800 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:21.439193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:52995 (local address 127.6.236.190:35033)
0925 01:24:21.439643 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:21.439650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:21.441035 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 01:24:21.443165 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:21.443169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:21.443234 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:21.443324 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:21.444629 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.445342 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.447296 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.447680 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.447770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:21.449152 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:21.449156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:21.449159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:21.449177 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:21.449217 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:21.449229 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:21.449293 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:21.449492 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 01:24:21.449566 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":209,"threads_started":1}
I20260925 01:24:21.450997 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:21.439798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35033 (local address 127.6.236.129:52995)
0925 01:24:21.440863 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:21.440924 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:21.442882 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:21.443577 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:21.443588 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:21.443697 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:21.444467 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:21.444491 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.445487 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.445492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:21.447168 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:21.447191 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.447873 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.447880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:21.448149 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:21.448835 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:21.448839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:21.448888 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:21.448981 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:21.449306 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:21.449315 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:21.449322 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:24:21.449477 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 01:24:21.449837 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":283,"threads_started":1}
I20260925 01:24:21.452472 16608 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35033
I20260925 01:24:21.452747 16608 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:21.453501 16608 heartbeater.cc:511] Master 127.6.236.190:35033 requested a full tablet report, sending...
I20260925 01:24:21.455646 16422 ts_manager.cc:194] Registered new tserver with Master: 76bac2a814134a09986a3933c7d549c2 (127.6.236.129:40979)
I20260925 01:24:21.457544 16422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.129:52995
W20260925 01:24:21.753659 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:21.753985 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:21.754078 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:21.761292 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:21.761380 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:21.761502 16612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:24:21.770304 16612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:21.771466 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:21.773164 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:21.785547 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:21.785643 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:21.785832 16620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:21.786629 16612 server_base.cc:1034] running on GCE node
I20260925 01:24:21.787159 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:21.788045 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:21.789251 16612 hybrid_clock.cc:648] HybridClock initialized: now 1790299461789203 us; error 37 us; skew 500 ppm
I20260925 01:24:21.789520 16612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:21.791636 16612 webserver.cc:460] Webserver started at http://127.6.236.130:35345/ using document root <none> and password file <none>
I20260925 01:24:21.792256 16612 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:21.792338 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:21.792593 16612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:21.795327 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "f0234040b72742cca30735a0eb5beb28"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "aa100627b943c54ebabdf9e47770d9e9"
server_key_iv: "266897d0799077eaa85e936727b43a9c"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.796049 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "f0234040b72742cca30735a0eb5beb28"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "aa100627b943c54ebabdf9e47770d9e9"
server_key_iv: "266897d0799077eaa85e936727b43a9c"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.800778 16612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:24:21.803949 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.805176 16612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 01:24:21.805322 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "f0234040b72742cca30735a0eb5beb28"
format_stamp: "Formatted at 2026-09-25 01:24:21 on dist-test-slave-s0b2"
server_key: "aa100627b943c54ebabdf9e47770d9e9"
server_key_iv: "266897d0799077eaa85e936727b43a9c"
server_key_version: "encryptionkey@0"
I20260925 01:24:21.805508 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:21.827186 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:21.827960 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:21.828212 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:21.829063 16612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:21.830508 16612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:21.830592 16612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.830679 16612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:21.830732 16612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:21.859659 16612 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:45415
I20260925 01:24:21.859728 16738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:45415 every 8 connection(s)
I20260925 01:24:21.860882 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:24:21.866559 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16612
I20260925 01:24:21.866730 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:24:21.867267 7090 external_mini_cluster.cc:1428] Setting key 803a2c0d9369ef649097d3ce5d5af3c3
I20260925 01:24:21.870193 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--builtin_ntp_servers=127.6.236.148:37173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:24:21.873950 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:21.863309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:58065 (local address 127.6.236.190:35033)
0925 01:24:21.863503 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:21.863509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:21.864874 (+ 1365us) server_negotiation.cc:407] Connection header received
0925 01:24:21.866749 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:21.866754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:21.866838 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:21.866918 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:21.868202 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.869217 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.871389 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.871787 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.871881 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:21.873251 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:21.873256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:21.873259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:21.873277 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:21.873309 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:21.873321 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:21.873386 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:21.873626 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 01:24:21.873709 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:24:21.875129 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:21.863879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35033 (local address 127.6.236.130:58065)
0925 01:24:21.864730 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:21.864771 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:21.866425 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:21.867197 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:21.867208 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:21.867326 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:21.868037 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:21.868063 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.869366 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.869371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:21.871254 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:21.871270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:21.871988 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:21.871995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:21.872260 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:21.872928 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:21.872931 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:21.872991 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:21.873084 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:21.873398 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:21.873406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:21.873414 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:24:21.873573 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 01:24:21.873931 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260925 01:24:21.876398 16739 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35033
I20260925 01:24:21.876688 16739 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:21.877493 16739 heartbeater.cc:511] Master 127.6.236.190:35033 requested a full tablet report, sending...
I20260925 01:24:21.878893 16422 ts_manager.cc:194] Registered new tserver with Master: f0234040b72742cca30735a0eb5beb28 (127.6.236.130:45415)
I20260925 01:24:21.879671 16422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.130:58065
W20260925 01:24:22.186375 16742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:22.186651 16742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:22.186780 16742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:22.194083 16742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:22.194181 16742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:22.194347 16742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:24:22.203322 16742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:22.204567 16742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:22.206264 16742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:22.218484 16750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:22.218508 16748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:22.218726 16747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:22.219439 16742 server_base.cc:1034] running on GCE node
I20260925 01:24:22.220046 16742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:22.220922 16742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:22.222107 16742 hybrid_clock.cc:648] HybridClock initialized: now 1790299462222058 us; error 43 us; skew 500 ppm
I20260925 01:24:22.222414 16742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:22.224448 16742 webserver.cc:460] Webserver started at http://127.6.236.131:44185/ using document root <none> and password file <none>
I20260925 01:24:22.225067 16742 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:22.225147 16742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:22.225404 16742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:22.228070 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "823f7c0c8ba54976ba64a4d3918475c0"
format_stamp: "Formatted at 2026-09-25 01:24:22 on dist-test-slave-s0b2"
server_key: "6a27ea301d7ade017f2602fc70220597"
server_key_iv: "1e54cea9bb960530839e9921e40a6b15"
server_key_version: "encryptionkey@0"
I20260925 01:24:22.228770 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "823f7c0c8ba54976ba64a4d3918475c0"
format_stamp: "Formatted at 2026-09-25 01:24:22 on dist-test-slave-s0b2"
server_key: "6a27ea301d7ade017f2602fc70220597"
server_key_iv: "1e54cea9bb960530839e9921e40a6b15"
server_key_version: "encryptionkey@0"
I20260925 01:24:22.233709 16742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:24:22.236984 16756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:22.238276 16742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:24:22.238404 16742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "823f7c0c8ba54976ba64a4d3918475c0"
format_stamp: "Formatted at 2026-09-25 01:24:22 on dist-test-slave-s0b2"
server_key: "6a27ea301d7ade017f2602fc70220597"
server_key_iv: "1e54cea9bb960530839e9921e40a6b15"
server_key_version: "encryptionkey@0"
I20260925 01:24:22.238569 16742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:22.257530 16742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:22.258510 16742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:22.258800 16742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:22.259768 16742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:22.261197 16742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:22.261301 16742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:22.261409 16742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:22.261477 16742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:22.288228 16742 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44961
I20260925 01:24:22.288285 16868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44961 every 8 connection(s)
I20260925 01:24:22.289498 16742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:24:22.294086 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16742
I20260925 01:24:22.294296 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:24:22.294857 7090 external_mini_cluster.cc:1428] Setting key 400dc01a3750f42b550c28d65a082fbd
I20260925 01:24:22.302042 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:22.291930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:44669 (local address 127.6.236.190:35033)
0925 01:24:22.292100 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:22.292105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:22.293453 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 01:24:22.295726 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:22.295731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:22.295816 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:22.295902 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:22.297086 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.297900 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.299536 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.299984 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.300100 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:22.301276 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:22.301281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:22.301284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:22.301302 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:22.301332 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:22.301344 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:22.301417 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:22.301628 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:24:22.301723 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:24:22.303011 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:22.292436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35033 (local address 127.6.236.131:44669)
0925 01:24:22.293305 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:22.293341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:22.295463 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:22.296132 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:22.296141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:22.296254 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:22.296934 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:22.296953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.298041 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.298045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:22.299404 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:22.299417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.300198 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.300205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:22.300421 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:22.300984 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:22.300987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:22.301025 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:22.301100 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:22.301427 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:22.301438 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:22.301446 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:24:22.301605 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 01:24:22.301994 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":261,"threads_started":1}
I20260925 01:24:22.304255 16869 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35033
I20260925 01:24:22.304494 16869 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:22.305168 16869 heartbeater.cc:511] Master 127.6.236.190:35033 requested a full tablet report, sending...
I20260925 01:24:22.306403 16422 ts_manager.cc:194] Registered new tserver with Master: 823f7c0c8ba54976ba64a4d3918475c0 (127.6.236.131:44961)
I20260925 01:24:22.307147 16422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.131:44669
I20260925 01:24:22.309938 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:24:22.315490 16610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:22.314063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.6.236.190:35033)
0925 01:24:22.314217 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:22.314221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:22.314704 (+ 483us) server_negotiation.cc:407] Connection header received
0925 01:24:22.314828 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:22.314831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:22.314899 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:22.314976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:22.315283 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59954: BlockingRecv error: recv got EOF from 127.0.0.1:59954 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:24:22.316254 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16481
I20260925 01:24:22.323180 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:22.317047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.6.236.129:40979)
0925 01:24:22.317529 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:22.317536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:22.317559 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:24:22.317611 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:22.317617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:22.317794 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:22.317905 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:22.318206 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.319203 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.320403 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.320943 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.321029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:22.321894 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:22.321902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:22.321913 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:22.322505 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:22.322549 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:22.322568 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:22.322659 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:22.322761 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 01:24:22.322861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260925 01:24:22.324331 16603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:22.403365 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16481
I20260925 01:24:22.417191 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16612
I20260925 01:24:22.424705 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:22.418026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.6.236.130:45415)
0925 01:24:22.418560 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:22.418567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:22.418596 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:24:22.418654 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:22.418660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:22.418876 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:22.418999 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:22.419361 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.420347 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.421538 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.422121 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.422217 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:22.423317 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:22.423325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:22.423340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:22.423951 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:22.424001 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:22.424021 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:22.424124 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:22.424263 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 01:24:22.424376 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":275,"threads_started":1}
I20260925 01:24:22.425915 16734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:22.505147 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16612
I20260925 01:24:22.518360 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16742
I20260925 01:24:22.526337 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:22.519272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.6.236.131:44961)
0925 01:24:22.519937 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:22.519944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:22.519973 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:24:22.520036 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:22.520042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:22.520268 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:22.520407 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:22.520864 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.521903 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.523226 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:22.523808 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:22.523927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:22.524923 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:22.524931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:22.524945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:22.525515 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:22.525561 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:22.525581 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:22.525682 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:22.525834 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 01:24:22.525981 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":337,"threads_started":1}
I20260925 01:24:22.527522 16864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:22.607000 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16742
I20260925 01:24:22.620471 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16391
I20260925 01:24:22.621592 16452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:22.722488 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16391
2026-09-25T01:24:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2320 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T01:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:24:22Z Disabled control of system clock
I20260925 01:24:22.786062 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46883
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:24:23.095829 16888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:23.096095 16888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:23.096213 16888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:23.103252 16888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:24:23.103345 16888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:23.103400 16888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:23.103472 16888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:24:23.103526 16888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:24:23.112434 16888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:23.113667 16888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:23.115433 16888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:23.126303 16893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:23.126271 16894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:23.126324 16896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:23.127434 16888 server_base.cc:1034] running on GCE node
I20260925 01:24:23.128159 16888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:23.129448 16888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:23.130635 16888 hybrid_clock.cc:648] HybridClock initialized: now 1790299463130588 us; error 41 us; skew 500 ppm
I20260925 01:24:23.130957 16888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:23.133080 16888 webserver.cc:460] Webserver started at http://127.6.236.190:38433/ using document root <none> and password file <none>
I20260925 01:24:23.133734 16888 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:23.133831 16888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:23.134120 16888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:23.137038 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "381d789c085c45278b9a8b1c9b0118a0"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "45e454034a1dd7b8808460a07408f08a"
server_key_iv: "71b3a186d6a30432310c3f03a2c4b039"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.137871 16888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "381d789c085c45278b9a8b1c9b0118a0"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "45e454034a1dd7b8808460a07408f08a"
server_key_iv: "71b3a186d6a30432310c3f03a2c4b039"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.142664 16888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 01:24:23.145749 16902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:23.147042 16888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:24:23.147193 16888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "381d789c085c45278b9a8b1c9b0118a0"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "45e454034a1dd7b8808460a07408f08a"
server_key_iv: "71b3a186d6a30432310c3f03a2c4b039"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.147373 16888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:23.152753 16888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:23.153452 16888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:23.153714 16888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:23.170100 16888 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46883
I20260925 01:24:23.170137 16953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46883 every 8 connection(s)
I20260925 01:24:23.171336 16888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:24:23.173009 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16888
I20260925 01:24:23.173210 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:24:23.173843 7090 external_mini_cluster.cc:1428] Setting key 6fce7e296037fd92aaae4a8a5e22daa0
I20260925 01:24:23.175277 16954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:23.182523 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:23.175719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.6.236.190:46883)
0925 01:24:23.176777 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:23.176800 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:23.176852 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:24:23.178599 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:23.178625 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:23.179154 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:23.179573 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:23.180548 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:23.180555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:23.180572 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:23.180588 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:23.180676 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:23.180698 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:23.180917 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:23.181066 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 01:24:23.181326 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":333,"threads_started":1}
I20260925 01:24:23.189214 16954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0: Bootstrap starting.
I20260925 01:24:23.192960 16954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:23.193879 16954 log.cc:824] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:23.196368 16954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0: No bootstrap required, opened a new log
I20260925 01:24:23.203267 16954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } }
I20260925 01:24:23.203583 16954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:23.203653 16954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381d789c085c45278b9a8b1c9b0118a0, State: Initialized, Role: FOLLOWER
I20260925 01:24:23.204205 16954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } }
I20260925 01:24:23.204454 16954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:24:23.204591 16954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:24:23.204751 16954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:23.206786 16954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } }
I20260925 01:24:23.207229 16954 leader_election.cc:302] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 381d789c085c45278b9a8b1c9b0118a0; no voters:
I20260925 01:24:23.207759 16954 leader_election.cc:288] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:24:23.207875 16959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:24:23.208524 16959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [term 1 LEADER]: Becoming Leader. State: Replica: 381d789c085c45278b9a8b1c9b0118a0, State: Running, Role: LEADER
I20260925 01:24:23.208997 16959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } }
I20260925 01:24:23.209631 16954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:24:23.211642 16961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 381d789c085c45278b9a8b1c9b0118a0. Latest consensus state: current_term: 1 leader_uuid: "381d789c085c45278b9a8b1c9b0118a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } } }
I20260925 01:24:23.212301 16961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:23.211614 16960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "381d789c085c45278b9a8b1c9b0118a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d789c085c45278b9a8b1c9b0118a0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46883 } } }
I20260925 01:24:23.212878 16960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:23.219080 16965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:24:23.223670 16965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:24:23.231453 16965 catalog_manager.cc:1347] Generated new cluster ID: 355439be20ef4803b36b88e35a8a50be
I20260925 01:24:23.231565 16965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:24:23.242761 16965 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:24:23.243141 16965 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:24:23.253866 16965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 381d789c085c45278b9a8b1c9b0118a0: Generated new TSK 0
I20260925 01:24:23.254665 16965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:24:23.269894 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--builtin_ntp_servers=127.6.236.148:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:24:23.583787 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:23.584052 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:23.584153 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:23.591274 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:23.591367 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:23.591495 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:24:23.600452 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:23.601698 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:23.603406 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:23.615471 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:23.615650 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:23.615682 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:23.616520 16978 server_base.cc:1034] running on GCE node
I20260925 01:24:23.617141 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:23.618167 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:23.619441 16978 hybrid_clock.cc:648] HybridClock initialized: now 1790299463619376 us; error 58 us; skew 500 ppm
I20260925 01:24:23.619758 16978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:23.621870 16978 webserver.cc:460] Webserver started at http://127.6.236.129:35829/ using document root <none> and password file <none>
I20260925 01:24:23.622555 16978 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:23.622648 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:23.622951 16978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:23.627200 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "2ba9ffa0643e408f8117d3df2c1e07d7"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "3ebfa7669a79e82b460f22c0b8bd2979"
server_key_iv: "1521512135955c931765d7c5b5486368"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.627954 16978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "2ba9ffa0643e408f8117d3df2c1e07d7"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "3ebfa7669a79e82b460f22c0b8bd2979"
server_key_iv: "1521512135955c931765d7c5b5486368"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.632970 16978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 01:24:23.636179 16992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:23.637374 16978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:24:23.637519 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "2ba9ffa0643e408f8117d3df2c1e07d7"
format_stamp: "Formatted at 2026-09-25 01:24:23 on dist-test-slave-s0b2"
server_key: "3ebfa7669a79e82b460f22c0b8bd2979"
server_key_iv: "1521512135955c931765d7c5b5486368"
server_key_version: "encryptionkey@0"
I20260925 01:24:23.637718 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:23.643100 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:23.643837 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:23.644081 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:23.644912 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:23.646205 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:23.646337 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:23.646440 16978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:23.646507 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:23.675278 16978 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42063
I20260925 01:24:23.675339 17104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42063 every 8 connection(s)
I20260925 01:24:23.676499 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:24:23.679816 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 16978
I20260925 01:24:23.680025 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:24:23.680800 7090 external_mini_cluster.cc:1428] Setting key 14958d4cb053c2016c2508ea92970353
I20260925 01:24:23.683768 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--builtin_ntp_servers=127.6.236.148:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 01:24:23.684863 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:23.678945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:45173 (local address 127.6.236.190:46883)
0925 01:24:23.679109 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:23.679114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:23.680630 (+ 1516us) server_negotiation.cc:407] Connection header received
0925 01:24:23.682613 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:23.682618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:23.682680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:23.682782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:23.684271 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:23.684276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:23.684279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:23.684296 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:23.684329 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:23.684341 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:23.684437 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:23.684620 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 01:24:23.684665 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:24:23.686075 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:23.679400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46883 (local address 127.6.236.129:45173)
0925 01:24:23.680456 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:23.680503 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:23.682352 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:23.683014 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:23.683025 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:23.683170 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:23.683866 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:23.683870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:23.683955 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:23.684086 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:23.684430 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:23.684440 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:23.684447 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:24:23.684706 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 01:24:23.684962 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":276,"threads_started":1}
I20260925 01:24:23.687420 17105 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46883
I20260925 01:24:23.687688 17105 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:23.688256 17105 heartbeater.cc:511] Master 127.6.236.190:46883 requested a full tablet report, sending...
I20260925 01:24:23.690299 16919 ts_manager.cc:194] Registered new tserver with Master: 2ba9ffa0643e408f8117d3df2c1e07d7 (127.6.236.129:42063)
W20260925 01:24:23.989936 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:23.990190 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:23.990319 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:23.997391 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:23.997484 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:23.997618 17108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:24:24.006412 17108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:24.007635 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:24.009338 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:24.021500 17113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:24.021453 17116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:24.021538 17114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:24.022295 17108 server_base.cc:1034] running on GCE node
I20260925 01:24:24.023005 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:24.023988 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:24.025192 17108 hybrid_clock.cc:648] HybridClock initialized: now 1790299464025123 us; error 51 us; skew 500 ppm
I20260925 01:24:24.025473 17108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:24.027595 17108 webserver.cc:460] Webserver started at http://127.6.236.130:33917/ using document root <none> and password file <none>
I20260925 01:24:24.028232 17108 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:24.028360 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:24.028692 17108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:24.031406 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "0950c5419c24481c9a420c77bea7ae11"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "c8592353954586ca017e86eb6cbc5ada"
server_key_iv: "f8126e062aeaf54da7622dbb9c2aedb7"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.032140 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "0950c5419c24481c9a420c77bea7ae11"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "c8592353954586ca017e86eb6cbc5ada"
server_key_iv: "f8126e062aeaf54da7622dbb9c2aedb7"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.036830 17108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:24:24.040028 17122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.041309 17108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:24:24.041452 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "0950c5419c24481c9a420c77bea7ae11"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "c8592353954586ca017e86eb6cbc5ada"
server_key_iv: "f8126e062aeaf54da7622dbb9c2aedb7"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.041651 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:24.047003 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:24.047716 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:24.047950 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:24.048892 17108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:24.050562 17108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:24.050661 17108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.050773 17108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:24.050843 17108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.079639 17108 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:45855
I20260925 01:24:24.079694 17234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:45855 every 8 connection(s)
I20260925 01:24:24.080900 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:24:24.084586 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17108
I20260925 01:24:24.084776 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:24:24.085330 7090 external_mini_cluster.cc:1428] Setting key e2730979bf6face02b54acc1469670f0
I20260925 01:24:24.088109 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--builtin_ntp_servers=127.6.236.148:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 01:24:24.089107 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.083270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:56465 (local address 127.6.236.190:46883)
0925 01:24:24.083443 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.083447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.084961 (+ 1514us) server_negotiation.cc:407] Connection header received
0925 01:24:24.086995 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.087001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.087063 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.087179 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.088438 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:24.088444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:24.088448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:24.088466 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:24.088504 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:24.088518 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:24.088628 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:24.088777 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 01:24:24.088837 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:24:24.089927 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.083766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46883 (local address 127.6.236.130:56465)
0925 01:24:24.084797 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:24.084847 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:24.086732 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:24.087411 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:24.087419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:24.087512 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:24.088132 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:24.088134 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:24.088170 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:24.088269 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:24.088600 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:24.088605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:24.088610 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:24:24.088823 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 01:24:24.089015 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":249,"threads_started":1}
I20260925 01:24:24.091300 17235 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46883
I20260925 01:24:24.091571 17235 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:24.092131 17235 heartbeater.cc:511] Master 127.6.236.190:46883 requested a full tablet report, sending...
I20260925 01:24:24.093520 16919 ts_manager.cc:194] Registered new tserver with Master: 0950c5419c24481c9a420c77bea7ae11 (127.6.236.130:45855)
W20260925 01:24:24.393827 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:24.394105 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:24.394199 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:24.401206 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:24.401296 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:24.401408 17238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:24:24.410320 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:24.411582 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:24.413265 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:24.425415 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:24.425455 17244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:24.425480 17246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:24.426226 17238 server_base.cc:1034] running on GCE node
I20260925 01:24:24.426813 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:24.427716 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:24.428938 17238 hybrid_clock.cc:648] HybridClock initialized: now 1790299464428864 us; error 65 us; skew 500 ppm
I20260925 01:24:24.429247 17238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:24:24.431447 17238 webserver.cc:460] Webserver started at http://127.6.236.131:39509/ using document root <none> and password file <none>
I20260925 01:24:24.432082 17238 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:24.432175 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:24.432440 17238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:24.435225 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "5af523ae1ad541f2b832608d9252d03a"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "e9eaebf92ab7756d0edc9f4c0d9617ac"
server_key_iv: "2ffd72168c89332828360d7e9db382f1"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.435916 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "5af523ae1ad541f2b832608d9252d03a"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "e9eaebf92ab7756d0edc9f4c0d9617ac"
server_key_iv: "2ffd72168c89332828360d7e9db382f1"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.440793 17238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:24:24.444183 17252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.445518 17238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:24:24.445647 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "5af523ae1ad541f2b832608d9252d03a"
format_stamp: "Formatted at 2026-09-25 01:24:24 on dist-test-slave-s0b2"
server_key: "e9eaebf92ab7756d0edc9f4c0d9617ac"
server_key_iv: "2ffd72168c89332828360d7e9db382f1"
server_key_version: "encryptionkey@0"
I20260925 01:24:24.445840 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:24.451121 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:24.451871 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:24.452092 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:24.452935 17238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:24.454221 17238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:24.454337 17238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.454442 17238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:24.454510 17238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:24.482717 17238 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:34351
I20260925 01:24:24.482769 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:34351 every 8 connection(s)
I20260925 01:24:24.483928 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:24:24.487457 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17238
I20260925 01:24:24.487646 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:24:24.488114 7090 external_mini_cluster.cc:1428] Setting key c3c0c1d3009d5f4724f6b56627bc3d86
I20260925 01:24:24.491740 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.486260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:38351 (local address 127.6.236.190:46883)
0925 01:24:24.486434 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.486438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.487810 (+ 1372us) server_negotiation.cc:407] Connection header received
0925 01:24:24.489676 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.489682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.489749 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.489833 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.491063 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:24.491069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:24.491075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:24.491093 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:24.491125 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:24.491144 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:24.491248 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:24.491417 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 01:24:24.491480 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:24:24.492619 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.486754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46883 (local address 127.6.236.131:38351)
0925 01:24:24.487652 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:24.487690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:24.489442 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:24.490063 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:24.490071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:24.490160 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:24.490787 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:24:24.490790 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:24.490826 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:24:24.490908 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:24.491227 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:24.491237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:24.491244 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:24:24.491483 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 01:24:24.491747 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
I20260925 01:24:24.493800 17365 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46883
I20260925 01:24:24.494055 17365 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:24.494612 17365 heartbeater.cc:511] Master 127.6.236.190:46883 requested a full tablet report, sending...
I20260925 01:24:24.495818 16919 ts_manager.cc:194] Registered new tserver with Master: 5af523ae1ad541f2b832608d9252d03a (127.6.236.131:34351)
I20260925 01:24:24.502835 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:24:24.508256 16957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:24:24.506902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.6.236.190:46883)
0925 01:24:24.507078 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.507083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.507523 (+ 440us) server_negotiation.cc:407] Connection header received
0925 01:24:24.507641 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.507645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.507689 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.507762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.508025 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58952: BlockingRecv error: recv got EOF from 127.0.0.1:58952 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:24:24.508670 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16978
I20260925 01:24:24.511909 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.509590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.6.236.129:42063)
0925 01:24:24.510046 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.510052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.510076 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:24:24.510128 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.510135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.510372 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.510488 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.511311 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:24.511317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:24.511327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:24.511340 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:24.511382 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:24.511399 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:24.511495 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:24.511540 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 01:24:24.511618 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":198,"threads_started":1}
I20260925 01:24:24.512988 17100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:24.591173 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16978
I20260925 01:24:24.605706 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17108
I20260925 01:24:24.608969 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.606630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.6.236.130:45855)
0925 01:24:24.607151 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.607158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.607184 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:24:24.607246 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.607252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.607461 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.607586 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.608420 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:24.608425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:24.608437 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:24.608454 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:24.608493 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:24.608508 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:24.608607 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:24.608664 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 01:24:24.608739 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":233,"threads_started":1}
I20260925 01:24:24.610189 17230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:24.688911 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17108
I20260925 01:24:24.702385 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17238
I20260925 01:24:24.711325 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:24.703255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.6.236.131:34351)
0925 01:24:24.703876 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:24.703889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:24.703918 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:24:24.703981 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:24.703987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:24.704205 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:24.704346 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:24.710476 (+ 6130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:24.710486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:24.710504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:24:24.710537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:24.710595 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:24:24.710613 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:24.710789 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:24.710869 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 01:24:24.710976 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":306,"threads_started":1}
I20260925 01:24:24.712517 17360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:24.791841 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17238
I20260925 01:24:24.804834 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 16888
I20260925 01:24:24.805990 16949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:24.906616 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 16888
2026-09-25T01:24:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2185 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:24:24 dist-test-slave-s0b2 krb5kdc[17383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:24:24 dist-test-slave-s0b2 krb5kdc[17383](info): set up 2 sockets
Sep 25 01:24:24 dist-test-slave-s0b2 krb5kdc[17383](info): commencing operation
krb5kdc: starting...
W20260925 01:24:25.907572 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:24:26 dist-test-slave-s0b2 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:24:26.117094 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44929
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:26.425189 17396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:26.425485 17396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:26.425625 17396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:26.432749 17396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:24:26.432864 17396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:26.432942 17396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:26.433022 17396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:24:26.433081 17396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:24:26.442754 17396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:44929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:26.444022 17396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:26.445314 17396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:26.455971 17402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:26.456100 17401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:26.456009 17404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:26.457006 17396 server_base.cc:1034] running on GCE node
I20260925 01:24:26.457755 17396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:26.459200 17396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:26.460397 17396 hybrid_clock.cc:648] HybridClock initialized: now 1790299466460350 us; error 37 us; skew 500 ppm
I20260925 01:24:26.460712 17396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:26 dist-test-slave-s0b2 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:26.463689 17396 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:24:26.465111 17396 webserver.cc:460] Webserver started at http://127.6.236.190:41931/ using document root <none> and password file <none>
I20260925 01:24:26.465785 17396 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:26.465900 17396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:26.466190 17396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:26.469089 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "089e432437534eb3b1beecc76e2b1027"
format_stamp: "Formatted at 2026-09-25 01:24:26 on dist-test-slave-s0b2"
server_key: "a1b44a6c64eccdba394a6b744e38624f"
server_key_iv: "5d300b5d05fd1748b516d9e0929d9fa5"
server_key_version: "encryptionkey@0"
I20260925 01:24:26.469832 17396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "089e432437534eb3b1beecc76e2b1027"
format_stamp: "Formatted at 2026-09-25 01:24:26 on dist-test-slave-s0b2"
server_key: "a1b44a6c64eccdba394a6b744e38624f"
server_key_iv: "5d300b5d05fd1748b516d9e0929d9fa5"
server_key_version: "encryptionkey@0"
I20260925 01:24:26.474766 17396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:24:26.478005 17411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:26.479290 17396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:24:26.479444 17396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "089e432437534eb3b1beecc76e2b1027"
format_stamp: "Formatted at 2026-09-25 01:24:26 on dist-test-slave-s0b2"
server_key: "a1b44a6c64eccdba394a6b744e38624f"
server_key_iv: "5d300b5d05fd1748b516d9e0929d9fa5"
server_key_version: "encryptionkey@0"
I20260925 01:24:26.479612 17396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:26.495986 17396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:26.500159 17396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:26.500388 17396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:26.516858 17396 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:44929
I20260925 01:24:26.516887 17462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:44929 every 8 connection(s)
I20260925 01:24:26.518157 17396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:24:26.521771 17463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:26.525045 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17396
I20260925 01:24:26.525293 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:24:26.526062 7090 external_mini_cluster.cc:1428] Setting key 8b9e60464ec6e7901360415e64124865
I20260925 01:24:26.534170 17463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: Bootstrap starting.
I20260925 01:24:26.537763 17463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:26.539089 17463 log.cc:824] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:24:26 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:26.542164 17463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: No bootstrap required, opened a new log
I20260925 01:24:26.551432 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:26.528286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.6.236.190:44929)
0925 01:24:26.529162 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:26.529178 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:26.529563 (+ 385us) server_negotiation.cc:407] Connection header received
0925 01:24:26.532423 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:26.532455 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:26.532947 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:26.533393 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:26.534700 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:26.536648 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:26.537823 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:26.538846 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:26.539033 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:26.543109 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:26.543121 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:26.543149 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:26.543202 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:26.546955 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:26.547599 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:26.547607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:26.547617 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:26.547830 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:26.548147 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:26.548152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:26.548155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:26.548935 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:26.549263 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:26.549546 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 01:24:26.549998 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":375,"threads_started":1}
I20260925 01:24:26.552263 17463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } }
I20260925 01:24:26.552595 17463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:26.552716 17463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089e432437534eb3b1beecc76e2b1027, State: Initialized, Role: FOLLOWER
I20260925 01:24:26.553375 17463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } }
I20260925 01:24:26.553651 17463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:24:26.553785 17463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:24:26.553959 17463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:26.556197 17463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } }
I20260925 01:24:26.556620 17463 leader_election.cc:302] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089e432437534eb3b1beecc76e2b1027; no voters:
I20260925 01:24:26.557122 17463 leader_election.cc:288] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:24:26.557250 17468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:24:26.557888 17468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [term 1 LEADER]: Becoming Leader. State: Replica: 089e432437534eb3b1beecc76e2b1027, State: Running, Role: LEADER
I20260925 01:24:26.558409 17468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } }
I20260925 01:24:26.559106 17463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:24:26.560988 17469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089e432437534eb3b1beecc76e2b1027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } } }
I20260925 01:24:26.560998 17470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089e432437534eb3b1beecc76e2b1027. Latest consensus state: current_term: 1 leader_uuid: "089e432437534eb3b1beecc76e2b1027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e432437534eb3b1beecc76e2b1027" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 44929 } } }
I20260925 01:24:26.561415 17469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:26.561514 17470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027 [sys.catalog]: This master's current role is: LEADER
I20260925 01:24:26.564376 17473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:24:26.569137 17473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:24:26.578660 17473 catalog_manager.cc:1347] Generated new cluster ID: ae7483d866294c83809bda9aa481c241
I20260925 01:24:26.578764 17473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:24:26.604027 17473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:24:26.605351 17473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:24:26.614033 17473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: Generated new TSK 0
I20260925 01:24:26.614840 17473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:24:26.675875 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:27.021239 17489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:27.021517 17489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:27.021647 17489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:27.028651 17489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:27.028764 17489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:27.028939 17489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:24:27.038218 17489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:27.039429 17489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:27.040729 17489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:27.057480 17495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:27.057953 17494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:27.057521 17497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:27.060470 17489 server_base.cc:1034] running on GCE node
I20260925 01:24:27.061208 17489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:27.062453 17489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:27.063755 17489 hybrid_clock.cc:648] HybridClock initialized: now 1790299467063658 us; error 88 us; skew 500 ppm
I20260925 01:24:27.064394 17489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:27 dist-test-slave-s0b2 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:27.068177 17489 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:24:27.069929 17489 webserver.cc:460] Webserver started at http://127.6.236.129:46029/ using document root <none> and password file <none>
I20260925 01:24:27.070854 17489 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:27.071018 17489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:27.071364 17489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:27.075179 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "ce567702a3924179a92223084da1d7dd"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "7442be2c3063d088728c8f00d2fe1613"
server_key_iv: "5a38c4be9804759f65985393ec755c3a"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.076128 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "ce567702a3924179a92223084da1d7dd"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "7442be2c3063d088728c8f00d2fe1613"
server_key_iv: "5a38c4be9804759f65985393ec755c3a"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.082561 17489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 01:24:27.087329 17504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.089027 17489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 01:24:27.089241 17489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "ce567702a3924179a92223084da1d7dd"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "7442be2c3063d088728c8f00d2fe1613"
server_key_iv: "5a38c4be9804759f65985393ec755c3a"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.089464 17489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:27.112321 17489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:27.116114 17489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:27.116364 17489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:27.117233 17489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:27.118515 17489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:27.118593 17489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.118707 17489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:27.118759 17489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.147255 17489 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:43589
I20260925 01:24:27.147308 17616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:43589 every 8 connection(s)
I20260925 01:24:27.148497 17489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:24:27.149962 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17489
I20260925 01:24:27.150161 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:24:27.150872 7090 external_mini_cluster.cc:1428] Setting key 5e6894061a49faa258a6a52af8d43c39
Sep 25 01:24:27 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:27.169175 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:27.150978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:40029 (local address 127.6.236.190:44929)
0925 01:24:27.151464 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:27.151470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:27.152529 (+ 1059us) server_negotiation.cc:407] Connection header received
0925 01:24:27.154855 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:27.154859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:27.154931 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:27.155019 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:27.156834 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.157537 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.159586 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.159976 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.160068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:27.163707 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:27.163713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:27.163716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:27.163750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:27.166179 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:27.166990 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:27.166994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:27.166996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:27.167108 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:27.167515 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:27.167518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:27.167520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:27.167827 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:27.167934 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:27.168732 (+ 798us) server_negotiation.cc:299] Negotiation successful
0925 01:24:27.168915 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260925 01:24:27.170480 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:27.151529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44929 (local address 127.6.236.129:40029)
0925 01:24:27.152359 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:27.152412 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:27.154572 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:27.155285 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:27.155296 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:27.155958 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:27.156679 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:27.156709 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.157679 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.157683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:27.159455 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:27.159469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.160179 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.160194 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:27.160449 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:27.161087 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:27.161114 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:27.163475 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:27.166319 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:27.166327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:27.166343 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:27.166888 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:27.167196 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:27.167200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:27.167204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:27.167417 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:27.167940 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:27.167948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:27.168329 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 01:24:27.168595 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 01:24:27.169122 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":294,"threads_started":1}
I20260925 01:24:27.171983 17617 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44929
I20260925 01:24:27.172322 17617 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:27.173213 17617 heartbeater.cc:511] Master 127.6.236.190:44929 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:24:27.175362 17428 ts_manager.cc:194] Registered new tserver with Master: ce567702a3924179a92223084da1d7dd (127.6.236.129:43589)
I20260925 01:24:27.176783 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:40029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:24:27.204766 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:27.510722 17623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:27.510993 17623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:27.511124 17623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:27.518108 17623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:27.518227 17623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:27.518397 17623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:24:27.527782 17623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:27.528942 17623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:27.530179 17623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:27.542644 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:27.542675 17631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:27.542610 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:27.543326 17623 server_base.cc:1034] running on GCE node
I20260925 01:24:27.543900 17623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:27.544750 17623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:27.545933 17623 hybrid_clock.cc:648] HybridClock initialized: now 1790299467545888 us; error 38 us; skew 500 ppm
I20260925 01:24:27.546221 17623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:27 dist-test-slave-s0b2 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:27.549088 17623 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:24:27.550432 17623 webserver.cc:460] Webserver started at http://127.6.236.130:40965/ using document root <none> and password file <none>
I20260925 01:24:27.551084 17623 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:27.551198 17623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:27.551492 17623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:27.554219 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "ee5b492c338a40648973d6dcb8531590"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "a9d631269a236bba4a4de1c05bfbcb1e"
server_key_iv: "5b0c90c3c3305351561fbbd64ef0fc80"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.554987 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "ee5b492c338a40648973d6dcb8531590"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "a9d631269a236bba4a4de1c05bfbcb1e"
server_key_iv: "5b0c90c3c3305351561fbbd64ef0fc80"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.559731 17623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 01:24:27.562999 17638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.564239 17623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:24:27.564379 17623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "ee5b492c338a40648973d6dcb8531590"
format_stamp: "Formatted at 2026-09-25 01:24:27 on dist-test-slave-s0b2"
server_key: "a9d631269a236bba4a4de1c05bfbcb1e"
server_key_iv: "5b0c90c3c3305351561fbbd64ef0fc80"
server_key_version: "encryptionkey@0"
I20260925 01:24:27.564541 17623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:27.592368 17623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:27.596354 17623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:27.596611 17623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:27.597524 17623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:27.598907 17623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:27.598984 17623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.599095 17623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:27.599145 17623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:27.627585 17623 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:44095
I20260925 01:24:27.627625 17750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:44095 every 8 connection(s)
I20260925 01:24:27.628801 17623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:24:27.632787 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17623
I20260925 01:24:27.632992 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:24:27.633517 7090 external_mini_cluster.cc:1428] Setting key 83fc1b0cb00941906067cbea71d1e134
Sep 25 01:24:27 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:27.649037 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:27.631115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:59957 (local address 127.6.236.190:44929)
0925 01:24:27.631335 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:27.631341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:27.632430 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 01:24:27.634718 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:27.634722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:27.634787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:27.634865 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:27.636703 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.637384 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.639368 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.639773 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.639860 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:27.643614 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:27.643620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:27.643625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:27.643668 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:27.646116 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:27.646856 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:27.646859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:27.646862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:27.646989 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:27.647389 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:27.647393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:27.647395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:27.647671 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:27.647804 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:27.648482 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 01:24:27.648714 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:24:27.650035 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:27.631635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44929 (local address 127.6.236.130:59957)
0925 01:24:27.632279 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:27.632323 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:27.634438 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:27.635126 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:27.635137 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:27.635765 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:27.636537 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:27.636561 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.637531 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.637536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:27.639237 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:27.639256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:27.639966 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:27.639971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:27.640223 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:27.640866 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:27.640893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:27.643393 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:27.646275 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:27.646284 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:27.646302 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:27.646731 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:27.647088 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:27.647092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:27.647094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:27.647275 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:27.647793 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:27.647801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:27.648114 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 01:24:27.648345 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 01:24:27.648881 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":244,"threads_started":1}
I20260925 01:24:27.651646 17751 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44929
I20260925 01:24:27.651937 17751 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:27.652652 17751 heartbeater.cc:511] Master 127.6.236.190:44929 requested a full tablet report, sending...
I20260925 01:24:27.654332 17428 ts_manager.cc:194] Registered new tserver with Master: ee5b492c338a40648973d6dcb8531590 (127.6.236.130:44095)
I20260925 01:24:27.655453 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:59957
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:24:27.685060 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:24:27.993163 17756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:24:27.993444 17756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:24:27.993585 17756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:24:28.000736 17756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:24:28.000849 17756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:24:28.000994 17756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:24:28.010496 17756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:24:28.011737 17756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:24:28.013008 17756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:24:28.024950 17761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:28.025023 17764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:24:28.025000 17762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:24:28.025740 17756 server_base.cc:1034] running on GCE node
I20260925 01:24:28.026351 17756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:24:28.027185 17756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:24:28.028374 17756 hybrid_clock.cc:648] HybridClock initialized: now 1790299468028332 us; error 31 us; skew 500 ppm
I20260925 01:24:28.028685 17756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:24:28 dist-test-slave-s0b2 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:24:28.031533 17756 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:24:28.032793 17756 webserver.cc:460] Webserver started at http://127.6.236.131:37019/ using document root <none> and password file <none>
I20260925 01:24:28.033402 17756 fs_manager.cc:359] Metadata directory not provided
I20260925 01:24:28.033514 17756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:24:28.033784 17756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:24:28.036499 17756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "b7a10125ff5b4da7a3da87cfa320cbec"
format_stamp: "Formatted at 2026-09-25 01:24:28 on dist-test-slave-s0b2"
server_key: "38481100d60b9f3fdfa640dcc27fd08e"
server_key_iv: "cd258dff6b35050900c80b7c8b82a18a"
server_key_version: "encryptionkey@0"
I20260925 01:24:28.037213 17756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "b7a10125ff5b4da7a3da87cfa320cbec"
format_stamp: "Formatted at 2026-09-25 01:24:28 on dist-test-slave-s0b2"
server_key: "38481100d60b9f3fdfa640dcc27fd08e"
server_key_iv: "cd258dff6b35050900c80b7c8b82a18a"
server_key_version: "encryptionkey@0"
I20260925 01:24:28.041917 17756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 01:24:28.045086 17771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:28.046348 17756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:24:28.046481 17756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "b7a10125ff5b4da7a3da87cfa320cbec"
format_stamp: "Formatted at 2026-09-25 01:24:28 on dist-test-slave-s0b2"
server_key: "38481100d60b9f3fdfa640dcc27fd08e"
server_key_iv: "cd258dff6b35050900c80b7c8b82a18a"
server_key_version: "encryptionkey@0"
I20260925 01:24:28.046655 17756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:24:28.076625 17756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:24:28.080519 17756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:24:28.080783 17756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:24:28.081701 17756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:24:28.083055 17756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:24:28.083130 17756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:28.083246 17756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:24:28.083304 17756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:24:28.112329 17756 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:37245
I20260925 01:24:28.112360 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:37245 every 8 connection(s)
I20260925 01:24:28.113550 17756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:24:28.114764 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 17756
I20260925 01:24:28.114945 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:24:28.115334 7090 external_mini_cluster.cc:1428] Setting key 12623b2afc21b515f58c6af6e855faa4
Sep 25 01:24:28 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:24:28.133273 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.117346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:60281 (local address 127.6.236.190:44929)
0925 01:24:28.117547 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.117554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.118941 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 01:24:28.120667 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.120670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.120733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.120813 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:28.122208 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.122968 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.124396 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.124798 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.124886 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.127978 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:28.127983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:28.127986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:28.128019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:28.130898 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.131505 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.131512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.131514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.131620 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.131949 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.131952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.131954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.132192 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:28.132292 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.132884 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.133036 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:24:28.134200 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.118000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:44929 (local address 127.6.236.131:60281)
0925 01:24:28.118790 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.118841 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.120453 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.121000 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.121007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.121535 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:24:28.122081 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.122098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.123081 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.123084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.124294 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.124306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.124953 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.124958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.125146 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.125752 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:24:28.125773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:24:28.127778 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:24:28.131017 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:28.131023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:28.131036 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:28.131423 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:28.131704 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:24:28.131707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:24:28.131709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:24:28.131875 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:24:28.132276 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:24:28.132281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:24:28.132569 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.132812 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.133170 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":330,"threads_started":1}
I20260925 01:24:28.135399 17884 heartbeater.cc:347] Connected to a master server at 127.6.236.190:44929
I20260925 01:24:28.135643 17884 heartbeater.cc:464] Registering TS with master...
I20260925 01:24:28.136287 17884 heartbeater.cc:511] Master 127.6.236.190:44929 requested a full tablet report, sending...
I20260925 01:24:28.137542 17428 ts_manager.cc:194] Registered new tserver with Master: b7a10125ff5b4da7a3da87cfa320cbec (127.6.236.131:37245)
I20260925 01:24:28.138423 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:60281
I20260925 01:24:28.142462 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:24:28.158896 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.146519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.6.236.190:44929)
0925 01:24:28.146694 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.146700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.147289 (+ 589us) server_negotiation.cc:407] Connection header received
0925 01:24:28.147417 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.147422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.147500 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.147606 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:28.148931 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.150032 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.151820 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.152221 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.152314 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.154439 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:28.154444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:28.154447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:28.154478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:28.156783 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.157339 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.157342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.157344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.157458 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.157723 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.157726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.157728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.158007 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:28.158185 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.158476 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.158638 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:24:28.162648 17428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:24:28.169277 17428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:24:28.180123 17617 heartbeater.cc:503] Master 127.6.236.190:44929 was elected leader, sending a full tablet report...
I20260925 01:24:28.191965 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.181644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.6.236.130:44095)
0925 01:24:28.182166 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.182176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.182205 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:24:28.182339 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.182348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.182608 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.183531 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:24:28.183791 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.185408 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.188310 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.189846 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.191143 (+ 1297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.191367 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.191440 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.191623 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":606,"threads_started":1}
I20260925 01:24:28.192313 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.180462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:44095 (local address 127.0.0.1:39296)
0925 01:24:28.181243 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.181294 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.181469 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.182865 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.182874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.182907 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:24:28.183326 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.183339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.185569 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.185575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.188087 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.188104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.190078 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.190085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.190308 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.190367 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.190480 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.190670 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":504,"threads_started":1}
I20260925 01:24:28.197260 17686 tablet_service.cc:1467] Processing CreateTablet for tablet fc12d0731879400eaa44db0f7efb8645 (DEFAULT_TABLE table=test-table [id=8ba3b47b1272420ea2336615daa080db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:24:28.198798 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc12d0731879400eaa44db0f7efb8645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:28.205775 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.180795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43589 (local address 127.0.0.1:46570)
0925 01:24:28.192982 (+ 12187us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.193001 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.193096 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.200286 (+ 7190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.200290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.200316 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:24:28.200506 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.200517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.202291 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.202297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.203510 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.203522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.205017 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.205023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.205212 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.205245 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.205327 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.205459 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"mutex_wait_us":1,"thread_start_us":18780,"threads_started":1}
I20260925 01:24:28.205824 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.182100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.6.236.129:43589)
0925 01:24:28.199143 (+ 17043us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.199152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.199175 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:24:28.199258 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.199265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.199487 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.200281 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:24:28.200667 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.202128 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.203649 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.204867 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.204984 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.205202 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.205390 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.205533 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16886,"thread_start_us":16126,"threads_started":1}
I20260925 01:24:28.206897 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.181179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:37245 (local address 127.0.0.1:34354)
0925 01:24:28.198652 (+ 17473us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.198672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.198800 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.199662 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.199671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.199791 (+ 120us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:24:28.200053 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.200066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.202020 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.202025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.204305 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.204320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.206145 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.206152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.206358 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.206392 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.206469 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.206584 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17344,"thread_start_us":17558,"threads_started":1}
I20260925 01:24:28.207015 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.182155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.6.236.131:37245)
0925 01:24:28.195763 (+ 13608us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.195769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.198775 (+ 3006us) server_negotiation.cc:407] Connection header received
0925 01:24:28.198935 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.198942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.199179 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.199377 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:24:28.200236 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.201880 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.204488 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.205925 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.206050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.206274 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.206540 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.206694 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13470,"thread_start_us":16854,"threads_started":1}
I20260925 01:24:28.209897 17552 tablet_service.cc:1467] Processing CreateTablet for tablet fc12d0731879400eaa44db0f7efb8645 (DEFAULT_TABLE table=test-table [id=8ba3b47b1272420ea2336615daa080db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:24:28.209916 17819 tablet_service.cc:1467] Processing CreateTablet for tablet fc12d0731879400eaa44db0f7efb8645 (DEFAULT_TABLE table=test-table [id=8ba3b47b1272420ea2336615daa080db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:24:28.211189 17819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc12d0731879400eaa44db0f7efb8645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:28.211270 17552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc12d0731879400eaa44db0f7efb8645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:24:28.213323 17902 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Bootstrap starting.
I20260925 01:24:28.216859 17902 tablet_bootstrap.cc:654] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:28.218137 17902 log.cc:824] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:28.221047 17902 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: No bootstrap required, opened a new log
I20260925 01:24:28.221355 17902 ts_tablet_manager.cc:1397] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 01:24:28.222951 17904 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Bootstrap starting.
I20260925 01:24:28.224259 17905 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Bootstrap starting.
I20260925 01:24:28.226392 17904 tablet_bootstrap.cc:654] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:28.227311 17904 log.cc:824] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:28.227455 17905 tablet_bootstrap.cc:654] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Neither blocks nor log segments found. Creating new log.
I20260925 01:24:28.228582 17905 log.cc:824] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Log is configured to *not* fsync() on all Append() calls
I20260925 01:24:28.229928 17904 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: No bootstrap required, opened a new log
I20260925 01:24:28.230157 17904 ts_tablet_manager.cc:1397] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 01:24:28.231370 17905 tablet_bootstrap.cc:492] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: No bootstrap required, opened a new log
I20260925 01:24:28.231178 17902 raft_consensus.cc:348] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.231601 17902 raft_consensus.cc:374] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:28.231619 17905 ts_tablet_manager.cc:1397] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 01:24:28.231701 17902 raft_consensus.cc:729] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5b492c338a40648973d6dcb8531590, State: Initialized, Role: FOLLOWER
I20260925 01:24:28.232306 17902 consensus_queue.cc:260] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.233357 17751 heartbeater.cc:503] Master 127.6.236.190:44929 was elected leader, sending a full tablet report...
I20260925 01:24:28.235252 17902 ts_tablet_manager.cc:1428] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260925 01:24:28.238271 17904 raft_consensus.cc:348] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.238686 17904 raft_consensus.cc:374] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:28.238795 17904 raft_consensus.cc:729] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce567702a3924179a92223084da1d7dd, State: Initialized, Role: FOLLOWER
I20260925 01:24:28.239387 17904 consensus_queue.cc:260] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.240438 17905 raft_consensus.cc:348] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.240752 17905 raft_consensus.cc:374] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:24:28.240850 17905 raft_consensus.cc:729] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a10125ff5b4da7a3da87cfa320cbec, State: Initialized, Role: FOLLOWER
I20260925 01:24:28.240967 17904 ts_tablet_manager.cc:1428] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260925 01:24:28.241398 17905 consensus_queue.cc:260] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.242455 17884 heartbeater.cc:503] Master 127.6.236.190:44929 was elected leader, sending a full tablet report...
I20260925 01:24:28.242918 17905 ts_tablet_manager.cc:1428] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 01:24:28.310762 17910 raft_consensus.cc:482] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:24:28.311038 17910 raft_consensus.cc:504] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.312687 17910 leader_election.cc:288] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5b492c338a40648973d6dcb8531590 (127.6.236.130:44095), ce567702a3924179a92223084da1d7dd (127.6.236.129:43589)
I20260925 01:24:28.318709 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.313099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:47587 (local address 127.6.236.130:44095)
0925 01:24:28.313278 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.313283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.313303 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:24:28.313521 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.313526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.313591 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.313666 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:24:28.314104 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.315049 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.316836 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.317808 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.317917 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.317967 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.318341 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.318444 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:24:28.318723 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.312871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:44095 (local address 127.6.236.131:47587)
0925 01:24:28.313200 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.313220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.313346 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.313730 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.313734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.313763 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:24:28.313945 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.313957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.315188 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.315193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.316670 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.316682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.317957 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.317963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.318144 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.318180 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.318316 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.318445 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":22}
I20260925 01:24:28.319238 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.313298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50213 (local address 127.6.236.129:43589)
0925 01:24:28.313483 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.313488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.313915 (+ 427us) server_negotiation.cc:407] Connection header received
0925 01:24:28.314061 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.314066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.314131 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.314200 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:24:28.314744 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.315592 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.317397 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.318353 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.318489 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.318555 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.318902 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.319007 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:24:28.319865 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.313143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43589 (local address 127.6.236.131:50213)
0925 01:24:28.313821 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0925 01:24:28.313835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:24:28.313943 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:24:28.314228 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:24:28.314233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:24:28.314281 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:24:28.314599 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.314610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.315746 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.315750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.317251 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:24:28.317265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.318538 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.318545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:24:28.318715 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.318746 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:24:28.319407 (+ 661us) client_negotiation.cc:240] Negotiation successful
0925 01:24:28.319527 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260925 01:24:28.319832 17706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc12d0731879400eaa44db0f7efb8645" candidate_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5b492c338a40648973d6dcb8531590" is_pre_election: true
I20260925 01:24:28.320353 17706 raft_consensus.cc:2454] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a10125ff5b4da7a3da87cfa320cbec in term 0.
I20260925 01:24:28.320873 17572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc12d0731879400eaa44db0f7efb8645" candidate_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce567702a3924179a92223084da1d7dd" is_pre_election: true
I20260925 01:24:28.321039 17774 leader_election.cc:302] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a10125ff5b4da7a3da87cfa320cbec, ee5b492c338a40648973d6dcb8531590; no voters:
I20260925 01:24:28.321400 17910 raft_consensus.cc:2781] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:24:28.321442 17572 raft_consensus.cc:2454] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7a10125ff5b4da7a3da87cfa320cbec in term 0.
I20260925 01:24:28.321538 17910 raft_consensus.cc:482] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:24:28.321622 17910 raft_consensus.cc:3037] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:28.323818 17910 raft_consensus.cc:504] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.324355 17910 leader_election.cc:288] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [CANDIDATE]: Term 1 election: Requested vote from peers ee5b492c338a40648973d6dcb8531590 (127.6.236.130:44095), ce567702a3924179a92223084da1d7dd (127.6.236.129:43589)
I20260925 01:24:28.324712 17706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc12d0731879400eaa44db0f7efb8645" candidate_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5b492c338a40648973d6dcb8531590"
I20260925 01:24:28.324765 17572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc12d0731879400eaa44db0f7efb8645" candidate_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce567702a3924179a92223084da1d7dd"
I20260925 01:24:28.324880 17706 raft_consensus.cc:3037] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:28.324918 17572 raft_consensus.cc:3037] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:24:28.326996 17572 raft_consensus.cc:2454] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a10125ff5b4da7a3da87cfa320cbec in term 1.
I20260925 01:24:28.327001 17706 raft_consensus.cc:2454] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7a10125ff5b4da7a3da87cfa320cbec in term 1.
I20260925 01:24:28.327334 17775 leader_election.cc:302] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a10125ff5b4da7a3da87cfa320cbec, ce567702a3924179a92223084da1d7dd; no voters:
I20260925 01:24:28.327605 17910 raft_consensus.cc:2781] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:24:28.327983 17910 raft_consensus.cc:686] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 1 LEADER]: Becoming Leader. State: Replica: b7a10125ff5b4da7a3da87cfa320cbec, State: Running, Role: LEADER
I20260925 01:24:28.328444 17910 consensus_queue.cc:237] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } }
I20260925 01:24:28.332870 17427 catalog_manager.cc:5495] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec reported cstate change: term changed from 0 to 1, leader changed from <none> to b7a10125ff5b4da7a3da87cfa320cbec (127.6.236.131). New cstate: current_term: 1 leader_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7a10125ff5b4da7a3da87cfa320cbec" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37245 } health_report { overall_health: HEALTHY } } }
I20260925 01:24:28.348517 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.343842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.6.236.131:37245)
0925 01:24:28.344022 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.344026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.344175 (+ 149us) server_negotiation.cc:407] Connection header received
0925 01:24:28.344354 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.344357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.344426 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.344537 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:24:28.344807 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.345538 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.346798 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.347169 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.347255 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.347662 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:24:28.348122 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:24:28.348223 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.348277 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.348337 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:24:28.355949 17572 raft_consensus.cc:1261] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 1 FOLLOWER]: Refusing update from remote peer b7a10125ff5b4da7a3da87cfa320cbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:24:28.355962 17706 raft_consensus.cc:1261] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 1 FOLLOWER]: Refusing update from remote peer b7a10125ff5b4da7a3da87cfa320cbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:24:28.356866 17912 consensus_queue.cc:1035] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce567702a3924179a92223084da1d7dd" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:24:28.357182 17910 consensus_queue.cc:1035] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5b492c338a40648973d6dcb8531590" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:24:28.365017 17885 tablet.cc:2369] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:24:28.367705 17915 mvcc.cc:205] Tried to move back new op lower bound from 7333066622376882176 to 7333066622276882432. Current Snapshot: MvccSnapshot[applied={T|T < 7333066622376882176}]
I20260925 01:24:28.385341 17428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:24:28.385605 17428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:24:28.389272 17428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 089e432437534eb3b1beecc76e2b1027: Sending DeleteTablet for 3 replicas of tablet fc12d0731879400eaa44db0f7efb8645
I20260925 01:24:28.390453 17819 tablet_service.cc:1514] Processing DeleteTablet for tablet fc12d0731879400eaa44db0f7efb8645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:24:28 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34354
I20260925 01:24:28.390450 17552 tablet_service.cc:1514] Processing DeleteTablet for tablet fc12d0731879400eaa44db0f7efb8645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:24:28 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:46570
I20260925 01:24:28.390463 17686 tablet_service.cc:1514] Processing DeleteTablet for tablet fc12d0731879400eaa44db0f7efb8645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:24:28 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:39296
I20260925 01:24:28.392830 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17489
I20260925 01:24:28.403095 17926 tablet_replica.cc:330] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: stopping tablet replica
I20260925 01:24:28.403602 17926 raft_consensus.cc:2229] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:24:28.404400 17926 raft_consensus.cc:2258] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:24:28.406836 17926 ts_tablet_manager.cc:1905] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:24:28.414857 17926 ts_tablet_manager.cc:1918] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:24:28.415069 17926 log.cc:1196] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/fc12d0731879400eaa44db0f7efb8645
I20260925 01:24:28.415650 17926 ts_tablet_manager.cc:1939] T fc12d0731879400eaa44db0f7efb8645 P b7a10125ff5b4da7a3da87cfa320cbec: Deleting consensus metadata
I20260925 01:24:28.417667 17413 catalog_manager.cc:4841] TS b7a10125ff5b4da7a3da87cfa320cbec (127.6.236.131:37245): tablet fc12d0731879400eaa44db0f7efb8645 (table test-table [id=8ba3b47b1272420ea2336615daa080db]) successfully deleted
I20260925 01:24:28.417855 17925 tablet_replica.cc:330] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: stopping tablet replica
I20260925 01:24:28.418257 17925 raft_consensus.cc:2229] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:24:28.418817 17925 raft_consensus.cc:2258] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:24:28.419574 17927 tablet_replica.cc:330] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: stopping tablet replica
I20260925 01:24:28.420059 17927 raft_consensus.cc:2229] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:24:28.420897 17925 ts_tablet_manager.cc:1905] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:24:28.420969 17927 raft_consensus.cc:2258] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:24:28.423974 17927 ts_tablet_manager.cc:1905] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:24:28.428179 17925 ts_tablet_manager.cc:1918] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:24:28.428339 17925 log.cc:1196] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/fc12d0731879400eaa44db0f7efb8645
I20260925 01:24:28.428848 17925 ts_tablet_manager.cc:1939] T fc12d0731879400eaa44db0f7efb8645 P ce567702a3924179a92223084da1d7dd: Deleting consensus metadata
Sep 25 01:24:28 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:24:28.430706 17927 ts_tablet_manager.cc:1918] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:24:28.430837 17927 log.cc:1196] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/fc12d0731879400eaa44db0f7efb8645
I20260925 01:24:28.431324 17927 ts_tablet_manager.cc:1939] T fc12d0731879400eaa44db0f7efb8645 P ee5b492c338a40648973d6dcb8531590: Deleting consensus metadata
I20260925 01:24:28.431356 17415 catalog_manager.cc:4841] TS ce567702a3924179a92223084da1d7dd (127.6.236.129:43589): tablet fc12d0731879400eaa44db0f7efb8645 (table test-table [id=8ba3b47b1272420ea2336615daa080db]) successfully deleted
I20260925 01:24:28.432616 17414 catalog_manager.cc:4841] TS ee5b492c338a40648973d6dcb8531590 (127.6.236.130:44095): tablet fc12d0731879400eaa44db0f7efb8645 (table test-table [id=8ba3b47b1272420ea2336615daa080db]) successfully deleted
I20260925 01:24:28.437111 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.394528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.6.236.129:43589)
0925 01:24:28.394719 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.394725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.418875 (+ 24150us) server_negotiation.cc:407] Connection header received
0925 01:24:28.419176 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.419182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.419282 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.419393 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:28.421104 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.422309 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.424380 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.425048 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.425178 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.430353 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:28.430369 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:28.430386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:28.430432 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:28.434510 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.435152 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.435161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.435165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.435313 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.435616 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.435621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.435624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.436076 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:28.436384 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.436542 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.436757 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:24:28.437819 17612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:28.528220 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17489
I20260925 01:24:28.543874 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17623
Sep 25 01:24:28 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:24:28.559666 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.544720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.6.236.130:44095)
0925 01:24:28.544924 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.544931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.544974 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:24:28.545039 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.545045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.545129 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.545256 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:28.546379 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.547500 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.548648 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.549280 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.549399 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.553429 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:28.553449 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:28.553470 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:28.553510 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:28.557025 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.557606 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.557615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.557619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.557763 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.558058 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.558062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.558066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.558557 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:28.558829 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.559065 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.559289 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:24:28.560395 17746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:28.646433 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17623
I20260925 01:24:28.660881 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17756
Sep 25 01:24:28 dist-test-slave-s0b2 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:24:28.676671 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:24:28.661625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.6.236.131:37245)
0925 01:24:28.661809 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:24:28.661816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:24:28.661882 (+ 66us) server_negotiation.cc:407] Connection header received
0925 01:24:28.662070 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:24:28.662075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:24:28.662148 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:24:28.662258 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:24:28.663501 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.664336 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.665541 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:24:28.665996 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:24:28.666117 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:24:28.671043 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:24:28.671054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:24:28.671070 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:24:28.671112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:24:28.673893 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.674722 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.674729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.674731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.674873 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:24:28.675279 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:24:28.675282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:24:28.675284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:24:28.675621 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:24:28.675872 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 01:24:28.676188 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 01:24:28.676345 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:24:28.677469 17879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:28.796708 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17756
I20260925 01:24:28.812644 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 17396
I20260925 01:24:28.813735 17458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:24:28.921725 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 17396
2026-09-25T01:24:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4014 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:24:29 dist-test-slave-s0b2 krb5kdc[17935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:24:29 dist-test-slave-s0b2 krb5kdc[17935](info): set up 2 sockets
Sep 25 01:24:29 dist-test-slave-s0b2 krb5kdc[17935](info): commencing operation
krb5kdc: starting...
W20260925 01:24:29.927735 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:24:30 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:24:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangeradmin_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangeradmin_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangerlookup_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangerlookup_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
I20260925 01:24:30.217072 7090 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgybJNj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/postgres -l logfile start
2026-09-25 01:24:32.483 UTC [17965] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:24:32.483 UTC [17965] LOG: listening on IPv4 address "127.6.236.148", port 51807
2026-09-25 01:24:32.486 UTC [17965] LOG: listening on Unix socket "/tmp/.s.PGSQL.51807"
2026-09-25 01:24:32.506 UTC [17968] LOG: database system was shut down at 2026-09-25 01:24:31 UTC
2026-09-25 01:24:32.509 UTC [17965] LOG: database system is ready to accept connections
I20260925 01:24:33.399713 7090 mini_postgres.cc:96] Postgres bound to 51807
2026-09-25 01:24:33.416 UTC [17976] FATAL: database "slave" does not exist
127.6.236.148:51807 - accepting connections
I20260925 01:24:33.417275 7090 mini_ranger.cc:161] Starting Ranger...
I20260925 01:24:33.440940 7090 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:24:33.861172 7090 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:24:33.861297 7090 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin
2026-09-25 01:24:34,127 [I] DB FLAVOR :POSTGRES
2026-09-25 01:24:34,128 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:24:34,128 [I] Checking connection..
2026-09-25 01:24:34,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:24:34,379 [I] Checking connection passed.
2026-09-25 01:24:34,380 [I] --------- Verifying version history table ---------
2026-09-25 01:24:34,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:24:34,632 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:24:34,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:24:34,882 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:24:34,883 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:24:34,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:24:35.127 UTC [18040] WARNING: there is no transaction in progress
2026-09-25 01:24:35,132 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:24:35,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:24:35,378 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:24:35,378 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:24:35,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:24:35,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:24:35,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s0b2.c.gcp-upstream.internal','N') ;"
2026-09-25 01:24:36,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:24:36,328 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:24:36,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:24:36,572 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:24:36,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:24:36,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:24:37,059 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:24:37,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:24:37.944 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:37.959 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:37.974 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:37.987 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.001 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.020 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.071 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.077 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.091 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.100 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.110 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.122 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.131 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.136 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.143 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38.149 UTC [18148] WARNING: there is no transaction in progress
2026-09-25 01:24:38,709 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:24:38,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:24:38,935 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:24:38,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:24:39,181 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:24:39,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:24:39,435 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:24:39,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:24:39,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:24:39,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s0b2.c.gcp-upstream.internal';"
2026-09-25 01:24:40,139 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:24:40,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:24:40,387 [I] DB_PATCHES have already been applied
I20260925 01:24:40.393476 7090 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgybJNj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:24:40.393580 7090 mini_ranger.cc:193] Using host: 127.6.236.148
I20260925 01:24:40.407610 7090 mini_ranger.cc:227] Ranger admin URL: http://127.6.236.148:50591
Sep 25, 2026 1:24:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:24:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:24:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:24:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:24:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:24:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:24:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:24:41.429442 7090 mini_ranger.cc:160] Time spent starting Ranger: real 8.012s user 0.001s sys 0.038s
Sep 25, 2026 1:25:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:25:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:25:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:25:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:25:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:25:02.687752 7090 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangerkms_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/rangerkms_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.236.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.236.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:25:02 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:25:02.833843 7090 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:25:02.852243 7090 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:25:03.631934 7090 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:25:03.632164 7090 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms
I20260925 01:25:03.632299 7090 mini_ranger_kms.cc:227] Using postgres at 127.6.236.148:51807
2026-09-25 01:25:04,528 [I] DB FLAVOR :POSTGRES
2026-09-25 01:25:04,528 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:25:04,528 [I] Checking connection
2026-09-25 01:25:04,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:25:04,760 [I] connection success
2026-09-25 01:25:04,760 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:25:04,760 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:25:04,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:25:04,983 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:25:04,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:25:04,984 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:25:04,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:25:05,229 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:25:05.695878 7090 mini_ranger_kms.cc:310] Created kms service
I20260925 01:25:05.886611 7090 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:25:05.926980 7090 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:25:05 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:06.290619 7090 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:25:06.290926 7090 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:25:06.290973 7090 mini_ranger_kms.cc:240] Using host: 127.6.236.148
I20260925 01:25:06.312157 7090 mini_ranger_kms.cc:276] Ranger KMS PID: 18328
I20260925 01:25:06.312337 7090 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.236.148:34267
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:25:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 01:25:08.431849 7090 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.598s user 0.148s sys 0.325s
I20260925 01:25:08.432132 7090 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:25:08.432387 7090 mini_ranger_kms.cc:401] 127.6.236.148:34267/kms/v1/keys
Sep 25 01:25:08 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 01:25:08 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:25:09 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:25:11 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:25:11.556142 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:34069
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:34069
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:34267/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:25:11.885955 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:11.886224 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:11.886337 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:11.893582 18379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:25:11.893684 18379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:11.893736 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:11.893793 18379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:25:11.893839 18379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:25:11.903630 18379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.236.148:34267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:34069
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:34069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:11.904809 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:11.906041 18379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:11.917006 18384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:11.917268 18385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:11.917362 18387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:11.918258 18379 server_base.cc:1034] running on GCE node
I20260925 01:25:11.919078 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:11.920421 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:11.921644 18379 hybrid_clock.cc:648] HybridClock initialized: now 1790299511921602 us; error 31 us; skew 500 ppm
I20260925 01:25:11.921962 18379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:11 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:11.924963 18379 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:25:11.926421 18379 webserver.cc:460] Webserver started at http://127.6.236.190:42107/ using document root <none> and password file <none>
I20260925 01:25:11.927049 18379 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:11.927157 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:11.927433 18379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:11 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:11.987231 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "a26cffc2266e49de8cf429c65be428a1"
format_stamp: "Formatted at 2026-09-25 01:25:11 on dist-test-slave-s0b2"
server_key: "90bc0643c426525068df4dd63cef750c"
server_key_iv: "087dddfa212edd21b380ddc959a8863b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:11.988281 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "a26cffc2266e49de8cf429c65be428a1"
format_stamp: "Formatted at 2026-09-25 01:25:11 on dist-test-slave-s0b2"
server_key: "90bc0643c426525068df4dd63cef750c"
server_key_iv: "087dddfa212edd21b380ddc959a8863b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:11.995235 18379 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260925 01:25:12.015355 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:12.017071 18379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:25:12.017254 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "a26cffc2266e49de8cf429c65be428a1"
format_stamp: "Formatted at 2026-09-25 01:25:11 on dist-test-slave-s0b2"
server_key: "90bc0643c426525068df4dd63cef750c"
server_key_iv: "087dddfa212edd21b380ddc959a8863b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:12.017508 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:12.034428 18379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:12.039143 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:12.039459 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:12.057597 18379 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:34069
I20260925 01:25:12.057659 18445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:34069 every 8 connection(s)
I20260925 01:25:12.058950 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:25:12.061203 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 18379
I20260925 01:25:12.061378 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:25:12.062063 7090 external_mini_cluster.cc:1428] Setting key ba962c69ee0c787a42f567fc16c55f26
I20260925 01:25:12.062836 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:25:12.076207 18446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: Bootstrap starting.
Sep 25 01:25:12 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:12.079825 18446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: Neither blocks nor log segments found. Creating new log.
I20260925 01:25:12.081132 18446 log.cc:824] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:25:12.084306 18446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: No bootstrap required, opened a new log
I20260925 01:25:12.085273 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:12.064550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.6.236.190:34069)
0925 01:25:12.065299 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:12.065314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:12.065361 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:25:12.067448 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:12.067476 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:12.067939 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:12.068377 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:12.069435 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:12.071449 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:12.072688 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:12.073719 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:12.073895 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:12.077364 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:12.077375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:12.077394 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:12.077438 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:12.080981 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:12.081673 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:12.081682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:12.081692 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:12.081871 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:12.082335 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:12.082340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:12.082343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:12.082998 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:12.083288 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:12.083619 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:25:12.083997 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":311,"threads_started":1}
I20260925 01:25:12.092090 18446 raft_consensus.cc:348] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } }
I20260925 01:25:12.092504 18446 raft_consensus.cc:374] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:25:12.092623 18446 raft_consensus.cc:729] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cffc2266e49de8cf429c65be428a1, State: Initialized, Role: FOLLOWER
I20260925 01:25:12.093194 18446 consensus_queue.cc:260] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } }
I20260925 01:25:12.093408 18446 raft_consensus.cc:388] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:25:12.093529 18446 raft_consensus.cc:482] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:25:12.093667 18446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:25:12.095678 18446 raft_consensus.cc:504] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } }
I20260925 01:25:12.096123 18446 leader_election.cc:302] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a26cffc2266e49de8cf429c65be428a1; no voters:
I20260925 01:25:12.096673 18446 leader_election.cc:288] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:25:12.096783 18451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:25:12.097460 18451 raft_consensus.cc:686] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [term 1 LEADER]: Becoming Leader. State: Replica: a26cffc2266e49de8cf429c65be428a1, State: Running, Role: LEADER
I20260925 01:25:12.097941 18451 consensus_queue.cc:237] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } }
I20260925 01:25:12.098552 18446 sys_catalog.cc:564] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:25:12.100418 18453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a26cffc2266e49de8cf429c65be428a1. Latest consensus state: current_term: 1 leader_uuid: "a26cffc2266e49de8cf429c65be428a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } } }
I20260925 01:25:12.100783 18453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:25:12.101279 18452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a26cffc2266e49de8cf429c65be428a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cffc2266e49de8cf429c65be428a1" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 34069 } } }
I20260925 01:25:12.101619 18452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:25:12.101833 18457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:25:12.106583 18457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:25:12.107788 18446 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:25:12.124161 18457 catalog_manager.cc:1347] Generated new cluster ID: b2075ac5b02248619da1970ddace06f3
I20260925 01:25:12.124401 18457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:25:12.149425 18457 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:25:12.150966 18457 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:25:12.159444 18457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: Generated new TSK 0
I20260925 01:25:12.160343 18457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:25:12.854817 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:34267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:25:13.252341 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:13.252692 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:13.252884 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:13.264111 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:13.264266 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:13.264453 18491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:25:13.279629 18491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.236.148:34267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:13.281411 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:13.283193 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:13.299957 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:13.300065 18501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:13.300084 18502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:13.312175 18491 server_base.cc:1034] running on GCE node
I20260925 01:25:13.313025 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:13.314458 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:13.315788 18491 hybrid_clock.cc:648] HybridClock initialized: now 1790299513315670 us; error 102 us; skew 500 ppm
I20260925 01:25:13.316273 18491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:13.320568 18491 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:25:13.333735 18491 webserver.cc:460] Webserver started at http://127.6.236.129:36315/ using document root <none> and password file <none>
I20260925 01:25:13.334806 18491 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:13.335016 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:13.335404 18491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:13.360072 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "cf4c189f5d184029b847633082e2749d"
format_stamp: "Formatted at 2026-09-25 01:25:13 on dist-test-slave-s0b2"
server_key: "a78d6ec8f65142c64b919e6b2ddef0a7"
server_key_iv: "6ccabe93b8c990a7e9970bdbe75777e1"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:13.361269 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "cf4c189f5d184029b847633082e2749d"
format_stamp: "Formatted at 2026-09-25 01:25:13 on dist-test-slave-s0b2"
server_key: "a78d6ec8f65142c64b919e6b2ddef0a7"
server_key_iv: "6ccabe93b8c990a7e9970bdbe75777e1"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:13.369462 18491 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260925 01:25:13.414110 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:13.416255 18491 fs_manager.cc:725] Time spent opening block manager: real 0.030s user 0.004s sys 0.004s
I20260925 01:25:13.416465 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "cf4c189f5d184029b847633082e2749d"
format_stamp: "Formatted at 2026-09-25 01:25:13 on dist-test-slave-s0b2"
server_key: "a78d6ec8f65142c64b919e6b2ddef0a7"
server_key_iv: "6ccabe93b8c990a7e9970bdbe75777e1"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:13.416690 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:13.440128 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:13.444178 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:13.444480 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:13.445500 18491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:25:13.452854 18491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:25:13.453016 18491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:25:13.453229 18491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:25:13.453348 18491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:13.619166 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:43241 every 8 connection(s)
I20260925 01:25:13.619578 18491 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:43241
I20260925 01:25:13.621284 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:25:13.626729 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 18491
I20260925 01:25:13.626941 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:25:13.627767 7090 external_mini_cluster.cc:1428] Setting key 8da744e2dc7b68ec61bbb44107f4da8d
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:13.669731 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:13.630023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34069 (local address 127.6.236.129:39305)
0925 01:25:13.642916 (+ 12893us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:13.642983 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:13.645455 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:13.648746 (+ 3291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:13.648758 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:13.649374 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:25:13.650186 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:13.650215 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:13.652344 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:13.652349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:13.654386 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:13.654411 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:13.655676 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:13.655683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:13.655992 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:13.656790 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:25:13.656819 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:25:13.659723 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:25:13.663808 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:13.663817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:13.663836 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:13.664406 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:13.665441 (+ 1035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:13.665446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:13.665449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:13.665700 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:13.667056 (+ 1356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:25:13.667065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:25:13.667546 (+ 481us) client_negotiation.cc:769] Sending connection context
0925 01:25:13.667823 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 01:25:13.668332 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12519,"thread_start_us":12586,"threads_started":1}
I20260925 01:25:13.670804 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:13.630621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:39305 (local address 127.6.236.190:34069)
0925 01:25:13.642397 (+ 11776us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:13.642404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:13.648120 (+ 5716us) server_negotiation.cc:407] Connection header received
0925 01:25:13.648200 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:13.648206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:13.648326 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:13.648443 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:13.650429 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:13.652147 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:13.654558 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:13.655221 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:13.655326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:13.659985 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:13.659992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:13.659996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:13.660036 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:13.663649 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:13.664960 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:13.664966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:13.664971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:13.665292 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:13.666546 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:13.666552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:13.666555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:13.666913 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:13.667059 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:13.670210 (+ 3151us) server_negotiation.cc:299] Negotiation successful
0925 01:25:13.670465 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11627,"thread_start_us":13006,"threads_started":1}
I20260925 01:25:13.672310 18624 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34069
I20260925 01:25:13.672631 18624 heartbeater.cc:464] Registering TS with master...
I20260925 01:25:13.673512 18624 heartbeater.cc:511] Master 127.6.236.190:34069 requested a full tablet report, sending...
I20260925 01:25:13.676116 18411 ts_manager.cc:194] Registered new tserver with Master: cf4c189f5d184029b847633082e2749d (127.6.236.129:43241)
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:25:13.678041 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:39305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:25:13.709756 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:34267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:25:13 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299513, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
W20260925 01:25:14.123449 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:14.123955 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:14.124154 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:14.135260 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:14.135475 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:14.135712 18630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:25:14.151382 18630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.236.148:34267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:14.153254 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:14.155001 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:14.187793 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:14.190733 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:14.194505 18630 server_base.cc:1034] running on GCE node
W20260925 01:25:14.194936 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:14.195782 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:14.202021 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:14.203281 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790299514203176 us; error 102 us; skew 500 ppm
I20260925 01:25:14.203735 18630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:14.208138 18630 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:25:14.228565 18630 webserver.cc:460] Webserver started at http://127.6.236.130:33089/ using document root <none> and password file <none>
I20260925 01:25:14.229662 18630 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:14.229889 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:14.230357 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:14.255267 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "4055a9287a584adcb1cbd8f52a82cc24"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "90c663a7a9b503cb39703e46e39bae62"
server_key_iv: "35e0f6a4fa7f0eb3cd2dfa8456187a3b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.256557 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "4055a9287a584adcb1cbd8f52a82cc24"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "90c663a7a9b503cb39703e46e39bae62"
server_key_iv: "35e0f6a4fa7f0eb3cd2dfa8456187a3b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.271418 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.003s
I20260925 01:25:14.306322 18654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.308410 18630 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.009s
I20260925 01:25:14.308697 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "4055a9287a584adcb1cbd8f52a82cc24"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "90c663a7a9b503cb39703e46e39bae62"
server_key_iv: "35e0f6a4fa7f0eb3cd2dfa8456187a3b"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.309029 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:14.354879 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:14.360549 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:14.360954 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:14.368667 18630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:25:14.371281 18630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:25:14.371364 18630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.371528 18630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:25:14.371592 18630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.407964 18630 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:34171
I20260925 01:25:14.407984 18766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:34171 every 8 connection(s)
I20260925 01:25:14.409268 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:25:14.416294 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 18630
I20260925 01:25:14.416453 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:25:14.416942 7090 external_mini_cluster.cc:1428] Setting key baec498d839f29e1135a146cc9b18448
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:14.433404 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:14.411961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:46759 (local address 127.6.236.190:34069)
0925 01:25:14.412460 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:14.412466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:14.413399 (+ 933us) server_negotiation.cc:407] Connection header received
0925 01:25:14.415413 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:14.415420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:14.415508 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:14.415649 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:14.417623 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.418866 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.421378 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.422003 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.422135 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:14.426756 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:14.426764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:14.426770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:14.426810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:14.429991 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.430881 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.430886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.430890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.431022 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.431486 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.431490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.431494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.431843 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:14.431983 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:14.432841 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 01:25:14.433060 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260925 01:25:14.434597 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:14.412448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34069 (local address 127.6.236.130:46759)
0925 01:25:14.413221 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:14.413267 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:14.415119 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:14.415948 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:14.415960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:14.416533 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:25:14.417377 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:14.417410 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.419062 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.419068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:14.421214 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:14.421241 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.422272 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.422281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:14.422763 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:14.423670 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:25:14.423703 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:25:14.426499 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:25:14.430154 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:14.430164 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:14.430183 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:14.430770 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:14.431125 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:14.431130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:14.431133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:14.431376 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:14.431980 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:25:14.431989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:25:14.432425 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 01:25:14.432724 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 01:25:14.433260 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":296,"threads_started":1}
I20260925 01:25:14.436365 18767 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34069
I20260925 01:25:14.436705 18767 heartbeater.cc:464] Registering TS with master...
I20260925 01:25:14.437628 18767 heartbeater.cc:511] Master 127.6.236.190:34069 requested a full tablet report, sending...
I20260925 01:25:14.439435 18411 ts_manager.cc:194] Registered new tserver with Master: 4055a9287a584adcb1cbd8f52a82cc24 (127.6.236.130:34171)
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:25:14.440514 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:46759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:25:14.469242 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:34267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:25:14.686832 18624 heartbeater.cc:503] Master 127.6.236.190:34069 was elected leader, sending a full tablet report...
W20260925 01:25:14.790943 18773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:14.791198 18773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:14.791326 18773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:14.798816 18773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:14.798919 18773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:14.799077 18773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:25:14.809134 18773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.236.148:34267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:34069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:14.810405 18773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:14.811669 18773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:14.823778 18779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:14.823805 18781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:14.824110 18778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:14.824537 18773 server_base.cc:1034] running on GCE node
I20260925 01:25:14.825153 18773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:14.825966 18773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:14.827160 18773 hybrid_clock.cc:648] HybridClock initialized: now 1790299514827104 us; error 46 us; skew 500 ppm
I20260925 01:25:14.827440 18773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:14.830297 18773 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:25:14.831913 18773 webserver.cc:460] Webserver started at http://127.6.236.131:41633/ using document root <none> and password file <none>
I20260925 01:25:14.832578 18773 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:14.832693 18773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:14.832980 18773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:14.847945 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "22af027531e44843b701da7976c38eaf"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "ae3119cef7c855c05069c1912ddb0db5"
server_key_iv: "32625659adcab9d8c09eea009de956ce"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.848780 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "22af027531e44843b701da7976c38eaf"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "ae3119cef7c855c05069c1912ddb0db5"
server_key_iv: "32625659adcab9d8c09eea009de956ce"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.853747 18773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:25:14.868994 18788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.870752 18773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:25:14.870909 18773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "22af027531e44843b701da7976c38eaf"
format_stamp: "Formatted at 2026-09-25 01:25:14 on dist-test-slave-s0b2"
server_key: "ae3119cef7c855c05069c1912ddb0db5"
server_key_iv: "32625659adcab9d8c09eea009de956ce"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:14.871112 18773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:14.903100 18773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:14.906946 18773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:14.907209 18773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:14.908126 18773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:25:14.909411 18773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:25:14.909487 18773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.909614 18773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:25:14.909667 18773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:14.939502 18773 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44685
I20260925 01:25:14.939524 18900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44685 every 8 connection(s)
I20260925 01:25:14.940713 18773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:25:14.949584 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 18773
I20260925 01:25:14.949771 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:25:14.950171 7090 external_mini_cluster.cc:1428] Setting key 841b33e4dde27fea7a43ebbb07f1279f
Sep 25 01:25:14 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299514, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:14.959146 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:14.943087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:44893 (local address 127.6.236.190:34069)
0925 01:25:14.943534 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:14.943539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:14.944488 (+ 949us) server_negotiation.cc:407] Connection header received
0925 01:25:14.946221 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:14.946226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:14.946337 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:14.946434 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:14.947768 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.948457 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.949991 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.950449 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.950557 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:14.954028 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:14.954033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:14.954036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:14.954071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:14.956382 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.957019 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.957023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.957025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.957149 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.957490 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.957493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.957495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.957772 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:14.957870 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:14.958618 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 01:25:14.958880 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":206,"threads_started":1}
I20260925 01:25:14.960033 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:14.943635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:34069 (local address 127.6.236.131:44893)
0925 01:25:14.944334 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:14.944371 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:14.945965 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:14.946667 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:14.946676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:14.947061 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:25:14.947609 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:14.947626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.948608 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.948612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:14.949855 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:14.949871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.950670 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.950677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:14.950902 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:14.951550 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:25:14.951572 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:25:14.953810 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:25:14.956505 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:14.956511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:14.956522 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:14.956924 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:14.957231 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:14.957234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:14.957236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:14.957416 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:14.957914 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:25:14.957920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:25:14.958203 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 01:25:14.958481 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 01:25:14.958936 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260925 01:25:14.961340 18901 heartbeater.cc:347] Connected to a master server at 127.6.236.190:34069
I20260925 01:25:14.961683 18901 heartbeater.cc:464] Registering TS with master...
I20260925 01:25:14.962368 18901 heartbeater.cc:511] Master 127.6.236.190:34069 requested a full tablet report, sending...
I20260925 01:25:14.963733 18411 ts_manager.cc:194] Registered new tserver with Master: 22af027531e44843b701da7976c38eaf (127.6.236.131:44685)
I20260925 01:25:14.964649 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:44893
I20260925 01:25:14.965553 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:25:14.981190 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:14.969886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.6.236.190:34069)
0925 01:25:14.970071 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:14.970078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:14.970403 (+ 325us) server_negotiation.cc:407] Connection header received
0925 01:25:14.970533 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:14.970537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:14.970613 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:14.970690 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:14.972073 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.972796 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.973996 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:14.974472 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:14.974562 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:14.976763 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:14.976769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:14.976773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:14.976803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:14.979051 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.979667 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.979671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.979673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.979793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:14.980064 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:14.980067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:14.980068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:14.980310 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:14.980474 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:14.980736 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:25:14.980894 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:25:14.984827 18411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:25:14.991707 18411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:25:15.014928 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.005923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.6.236.130:34171)
0925 01:25:15.007436 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.007442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.007465 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:25:15.007536 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.007541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.007740 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.007906 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:25:15.008550 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.010013 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.012099 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.013518 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.013642 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.013834 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.014409 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.014579 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":1284,"threads_started":1}
I20260925 01:25:15.014928 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.005422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:34171 (local address 127.0.0.1:54542)
0925 01:25:15.006616 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.006632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:15.006722 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:15.007934 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:15.007942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:15.007971 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:25:15.008345 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.008358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.010178 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.010183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.011927 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.011940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.013975 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.013981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.014175 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.014220 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:25:15.014355 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:25:15.014609 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":1562,"threads_started":1}
I20260925 01:25:15.015944 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.003735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34330)
0925 01:25:15.005934 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.005957 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:15.006066 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:15.009104 (+ 3038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:15.009109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:15.009131 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:25:15.009560 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.009572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.011501 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.011510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.013352 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.013365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.015128 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.015135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.015301 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.015333 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:25:15.015396 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 01:25:15.015513 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"mutex_wait_us":63,"thread_start_us":784,"threads_started":1}
I20260925 01:25:15.016480 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.003575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44685 (local address 127.0.0.1:34668)
0925 01:25:15.004592 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.004639 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:15.004817 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:15.009381 (+ 4564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:15.009390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:15.009412 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:25:15.009929 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.009943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.011914 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.011918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.014023 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.014035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.015755 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.015761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.015928 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.015963 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:25:15.016048 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 01:25:15.016162 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":19,"thread_start_us":2820,"threads_started":1}
I20260925 01:25:15.016989 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.004333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.6.236.129:43241)
0925 01:25:15.008544 (+ 4211us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.008552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.008580 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:25:15.008681 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.008688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.008918 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.009097 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:25:15.009768 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.011356 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.013516 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.014901 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.016261 (+ 1360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.016454 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.016519 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.016681 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4042,"thread_start_us":3878,"threads_started":1}
I20260925 01:25:15.017249 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.003815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.6.236.131:44685)
0925 01:25:15.008860 (+ 5045us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.008865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.008890 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:25:15.008960 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.008964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.009175 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.010138 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:25:15.010413 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.011787 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.014205 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.015549 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.015668 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.015883 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.016778 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.016924 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4919,"thread_start_us":4766,"threads_started":1}
I20260925 01:25:15.019307 18702 tablet_service.cc:1467] Processing CreateTablet for tablet 875a440eb0474a6db21667aa65188115 (DEFAULT_TABLE table=test-table [id=ec41943b4e59463e899d59b67d827786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:25:15.020478 18702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875a440eb0474a6db21667aa65188115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:25:15.020677 18559 tablet_service.cc:1467] Processing CreateTablet for tablet 875a440eb0474a6db21667aa65188115 (DEFAULT_TABLE table=test-table [id=ec41943b4e59463e899d59b67d827786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:25:15.020679 18836 tablet_service.cc:1467] Processing CreateTablet for tablet 875a440eb0474a6db21667aa65188115 (DEFAULT_TABLE table=test-table [id=ec41943b4e59463e899d59b67d827786]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:25:15.022071 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875a440eb0474a6db21667aa65188115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:25:15.022073 18559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875a440eb0474a6db21667aa65188115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:25:15.032179 18920 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Bootstrap starting.
I20260925 01:25:15.035367 18920 tablet_bootstrap.cc:654] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Neither blocks nor log segments found. Creating new log.
I20260925 01:25:15.036285 18922 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Bootstrap starting.
I20260925 01:25:15.036661 18920 log.cc:824] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Log is configured to *not* fsync() on all Append() calls
I20260925 01:25:15.037667 18921 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Bootstrap starting.
I20260925 01:25:15.040086 18922 tablet_bootstrap.cc:654] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Neither blocks nor log segments found. Creating new log.
I20260925 01:25:15.040395 18921 tablet_bootstrap.cc:654] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Neither blocks nor log segments found. Creating new log.
I20260925 01:25:15.041301 18921 log.cc:824] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:25:15.041358 18922 log.cc:824] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:25:15.044268 18921 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: No bootstrap required, opened a new log
I20260925 01:25:15.044340 18922 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: No bootstrap required, opened a new log
I20260925 01:25:15.044332 18920 tablet_bootstrap.cc:492] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: No bootstrap required, opened a new log
I20260925 01:25:15.044508 18921 ts_tablet_manager.cc:1397] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:25:15.044584 18922 ts_tablet_manager.cc:1397] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 01:25:15.044584 18920 ts_tablet_manager.cc:1397] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260925 01:25:15.051193 18921 raft_consensus.cc:348] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.051506 18921 raft_consensus.cc:374] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:25:15.051612 18921 raft_consensus.cc:729] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22af027531e44843b701da7976c38eaf, State: Initialized, Role: FOLLOWER
I20260925 01:25:15.052141 18921 consensus_queue.cc:260] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.053143 18901 heartbeater.cc:503] Master 127.6.236.190:34069 was elected leader, sending a full tablet report...
I20260925 01:25:15.053642 18921 ts_tablet_manager.cc:1428] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260925 01:25:15.054527 18920 raft_consensus.cc:348] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.054904 18920 raft_consensus.cc:374] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:25:15.055004 18920 raft_consensus.cc:729] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4055a9287a584adcb1cbd8f52a82cc24, State: Initialized, Role: FOLLOWER
I20260925 01:25:15.055510 18920 consensus_queue.cc:260] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.055850 18922 raft_consensus.cc:348] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.056283 18922 raft_consensus.cc:374] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:25:15.056403 18922 raft_consensus.cc:729] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4c189f5d184029b847633082e2749d, State: Initialized, Role: FOLLOWER
I20260925 01:25:15.056505 18767 heartbeater.cc:503] Master 127.6.236.190:34069 was elected leader, sending a full tablet report...
I20260925 01:25:15.057050 18920 ts_tablet_manager.cc:1428] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 01:25:15.057026 18922 consensus_queue.cc:260] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.058521 18922 ts_tablet_manager.cc:1428] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260925 01:25:15.147625 18625 tablet.cc:2369] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:25:15.161262 18768 tablet.cc:2369] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:25:15.189464 18928 raft_consensus.cc:482] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:25:15.189826 18928 raft_consensus.cc:504] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.192040 18928 leader_election.cc:288] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22af027531e44843b701da7976c38eaf (127.6.236.131:44685), 4055a9287a584adcb1cbd8f52a82cc24 (127.6.236.130:34171)
W20260925 01:25:15.192850 18902 tablet.cc:2369] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:25:15.200435 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.193870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:56821 (local address 127.6.236.131:44685)
0925 01:25:15.194071 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.194079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.194100 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:25:15.194175 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.194181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.194283 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.194367 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:25:15.195216 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.196158 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.198509 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.199699 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.199851 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.199925 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.200132 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.200253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:25:15.200640 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.192265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44685 (local address 127.6.236.129:56821)
0925 01:25:15.193427 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.193447 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:15.193595 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:15.194549 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:15.194554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:15.194582 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:25:15.195034 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.195047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.196264 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.196267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.198345 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.198360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.199872 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.199877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.199992 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.200015 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:25:15.200097 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:25:15.200229 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":1,"spinlock_wait_cycles":38144,"thread_start_us":831,"threads_started":1}
I20260925 01:25:15.200947 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.192832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:40125 (local address 127.6.236.130:34171)
0925 01:25:15.193654 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.193661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.194651 (+ 990us) server_negotiation.cc:407] Connection header received
0925 01:25:15.194945 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.194950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.195035 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.195095 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:25:15.195716 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.196597 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.198810 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.200014 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.200142 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.200205 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.200595 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.200725 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661}
I20260925 01:25:15.201488 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.192319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:34171 (local address 127.6.236.129:40125)
0925 01:25:15.194560 (+ 2241us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.194576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:15.194781 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:15.195282 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:15.195286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:15.195311 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:25:15.195575 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.195582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.196726 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.196732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.198680 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:15.198690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.200228 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.200234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:15.200408 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.200441 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:25:15.201111 (+ 670us) client_negotiation.cc:240] Negotiation successful
0925 01:25:15.201229 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"mutex_wait_us":18,"thread_start_us":2357,"threads_started":1}
I20260925 01:25:15.201583 18856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875a440eb0474a6db21667aa65188115" candidate_uuid: "cf4c189f5d184029b847633082e2749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22af027531e44843b701da7976c38eaf" is_pre_election: true
I20260925 01:25:15.202071 18856 raft_consensus.cc:2454] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4c189f5d184029b847633082e2749d in term 0.
I20260925 01:25:15.202353 18722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875a440eb0474a6db21667aa65188115" candidate_uuid: "cf4c189f5d184029b847633082e2749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4055a9287a584adcb1cbd8f52a82cc24" is_pre_election: true
I20260925 01:25:15.202724 18513 leader_election.cc:302] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22af027531e44843b701da7976c38eaf, cf4c189f5d184029b847633082e2749d; no voters:
I20260925 01:25:15.202831 18722 raft_consensus.cc:2454] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4c189f5d184029b847633082e2749d in term 0.
I20260925 01:25:15.203123 18928 raft_consensus.cc:2781] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:25:15.203286 18928 raft_consensus.cc:482] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:25:15.203394 18928 raft_consensus.cc:3037] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:25:15.205588 18928 raft_consensus.cc:504] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.206144 18928 leader_election.cc:288] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [CANDIDATE]: Term 1 election: Requested vote from peers 22af027531e44843b701da7976c38eaf (127.6.236.131:44685), 4055a9287a584adcb1cbd8f52a82cc24 (127.6.236.130:34171)
I20260925 01:25:15.206575 18856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875a440eb0474a6db21667aa65188115" candidate_uuid: "cf4c189f5d184029b847633082e2749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22af027531e44843b701da7976c38eaf"
I20260925 01:25:15.206606 18722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875a440eb0474a6db21667aa65188115" candidate_uuid: "cf4c189f5d184029b847633082e2749d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4055a9287a584adcb1cbd8f52a82cc24"
I20260925 01:25:15.206739 18856 raft_consensus.cc:3037] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:25:15.206769 18722 raft_consensus.cc:3037] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:25:15.208781 18856 raft_consensus.cc:2454] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4c189f5d184029b847633082e2749d in term 1.
I20260925 01:25:15.208840 18722 raft_consensus.cc:2454] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4c189f5d184029b847633082e2749d in term 1.
I20260925 01:25:15.209153 18513 leader_election.cc:302] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22af027531e44843b701da7976c38eaf, cf4c189f5d184029b847633082e2749d; no voters:
I20260925 01:25:15.209434 18928 raft_consensus.cc:2781] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:25:15.209838 18928 raft_consensus.cc:686] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 1 LEADER]: Becoming Leader. State: Replica: cf4c189f5d184029b847633082e2749d, State: Running, Role: LEADER
I20260925 01:25:15.210354 18928 consensus_queue.cc:237] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } }
I20260925 01:25:15.214557 18409 catalog_manager.cc:5495] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d reported cstate change: term changed from 0 to 1, leader changed from <none> to cf4c189f5d184029b847633082e2749d (127.6.236.129). New cstate: current_term: 1 leader_uuid: "cf4c189f5d184029b847633082e2749d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf4c189f5d184029b847633082e2749d" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 } health_report { overall_health: UNKNOWN } } }
I20260925 01:25:15.252511 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.247679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.6.236.129:43241)
0925 01:25:15.247844 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.247848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.247864 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:25:15.247948 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.247952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.248025 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.248142 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:25:15.248530 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.249238 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.250636 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.251017 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.251105 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.251535 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:25:15.252017 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:25:15.252117 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.252212 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.252273 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:25:15.260351 18856 raft_consensus.cc:1261] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 1 FOLLOWER]: Refusing update from remote peer cf4c189f5d184029b847633082e2749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:25:15.260352 18722 raft_consensus.cc:1261] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 1 FOLLOWER]: Refusing update from remote peer cf4c189f5d184029b847633082e2749d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:25:15.261230 18931 consensus_queue.cc:1035] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "22af027531e44843b701da7976c38eaf" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:25:15.261595 18928 consensus_queue.cc:1035] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4055a9287a584adcb1cbd8f52a82cc24" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:25:15.278976 18936 mvcc.cc:205] Tried to move back new op lower bound from 7333066814497067008 to 7333066814304931840. Current Snapshot: MvccSnapshot[applied={T|T < 7333066814497067008}]
I20260925 01:25:15.298174 18410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:25:15.298549 18410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:25:15.302484 18410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a26cffc2266e49de8cf429c65be428a1: Sending DeleteTablet for 3 replicas of tablet 875a440eb0474a6db21667aa65188115
I20260925 01:25:15.303732 18559 tablet_service.cc:1514] Processing DeleteTablet for tablet 875a440eb0474a6db21667aa65188115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:25:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34330
I20260925 01:25:15.303877 18836 tablet_service.cc:1514] Processing DeleteTablet for tablet 875a440eb0474a6db21667aa65188115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:25:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34668
I20260925 01:25:15.304023 18702 tablet_service.cc:1514] Processing DeleteTablet for tablet 875a440eb0474a6db21667aa65188115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:25:15 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:54542
I20260925 01:25:15.306147 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 18491
I20260925 01:25:15.320585 18945 tablet_replica.cc:330] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: stopping tablet replica
I20260925 01:25:15.321799 18945 raft_consensus.cc:2229] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:25:15.322618 18944 tablet_replica.cc:330] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: stopping tablet replica
I20260925 01:25:15.323601 18945 raft_consensus.cc:2258] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:25:15.324035 18944 raft_consensus.cc:2229] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:25:15.324517 18944 raft_consensus.cc:2258] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:25:15.326912 18945 ts_tablet_manager.cc:1905] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:25:15.328347 18944 ts_tablet_manager.cc:1905] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:25:15.328922 18943 tablet_replica.cc:330] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: stopping tablet replica
I20260925 01:25:15.329252 18943 raft_consensus.cc:2229] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:25:15.329870 18943 raft_consensus.cc:2258] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 01:25:15 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:25:15.333806 18945 ts_tablet_manager.cc:1918] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:25:15.333967 18945 log.cc:1196] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/875a440eb0474a6db21667aa65188115
I20260925 01:25:15.334476 18945 ts_tablet_manager.cc:1939] T 875a440eb0474a6db21667aa65188115 P 4055a9287a584adcb1cbd8f52a82cc24: Deleting consensus metadata
I20260925 01:25:15.336395 18395 catalog_manager.cc:4841] TS 4055a9287a584adcb1cbd8f52a82cc24 (127.6.236.130:34171): tablet 875a440eb0474a6db21667aa65188115 (table test-table [id=ec41943b4e59463e899d59b67d827786]) successfully deleted
I20260925 01:25:15.336793 18943 ts_tablet_manager.cc:1905] T 875a440eb0474a6db21667aa65188115 P cf4c189f5d184029b847633082e2749d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:25:15.339303 18944 ts_tablet_manager.cc:1918] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:25:15.339471 18944 log.cc:1196] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/875a440eb0474a6db21667aa65188115
I20260925 01:25:15.339960 18944 ts_tablet_manager.cc:1939] T 875a440eb0474a6db21667aa65188115 P 22af027531e44843b701da7976c38eaf: Deleting consensus metadata
I20260925 01:25:15.340508 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.307683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.6.236.129:43241)
0925 01:25:15.307898 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.307908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.312851 (+ 4943us) server_negotiation.cc:407] Connection header received
0925 01:25:15.313757 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.313763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.313873 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.313942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:15.317909 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.319279 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.321805 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.322604 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.322717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.332560 (+ 9843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:15.332574 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:15.332591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:15.332637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:15.336753 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.337681 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.337690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.337693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.337868 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.338346 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.338351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.338353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.338811 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:15.339089 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.339877 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.340177 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 01:25:15.341760 18396 catalog_manager.cc:4841] TS 22af027531e44843b701da7976c38eaf (127.6.236.131:44685): tablet 875a440eb0474a6db21667aa65188115 (table test-table [id=ec41943b4e59463e899d59b67d827786]) successfully deleted
I20260925 01:25:15.342203 18619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:25:15.450358 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 18491
I20260925 01:25:15.466312 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 18630
W20260925 01:25:15.466455 18395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.236.129:43241 (error 108)
I20260925 01:25:15.468166 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.467743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34348)
0925 01:25:15.467929 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.468029 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 01:25:15.468402 18395 catalog_manager.cc:4584] TS cf4c189f5d184029b847633082e2749d (127.6.236.129:43241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 875a440eb0474a6db21667aa65188115: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Sep 25 01:25:15 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:25:15.480690 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.467082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.6.236.130:34171)
0925 01:25:15.467569 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.467575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.467589 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:25:15.467640 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.467643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.467701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.467778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:15.469169 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.469893 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.471204 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.471599 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.471705 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.475615 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:15.475631 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:15.475647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:15.475683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:15.478458 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.479070 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.479076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.479078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.479199 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.479464 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.479466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.479468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.479809 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:15.480016 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.480215 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.480365 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260925 01:25:15.481314 18762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:25:15.521826 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.521339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34360)
0925 01:25:15.521582 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.521681 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:25:15.522905 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.522590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34364)
0925 01:25:15.522710 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.522794 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 01:25:15.569000 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 18630
I20260925 01:25:15.583145 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 18773
I20260925 01:25:15.583340 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.582732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34368)
0925 01:25:15.583084 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.583157 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260925 01:25:15.584350 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.583962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34380)
0925 01:25:15.584123 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.584207 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
Sep 25 01:25:15 dist-test-slave-s0b2 krb5kdc[17935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:25:15.597262 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.583896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.6.236.131:44685)
0925 01:25:15.584075 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:15.584081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:15.584124 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:25:15.584314 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:15.584318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:15.584380 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:15.584469 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:15.585729 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.586457 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.587758 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:15.588115 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:15.588213 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:15.592380 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:15.592394 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:15.592410 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:15.592461 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:15.595078 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.595673 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.595680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.595683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.595800 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:15.596075 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:15.596078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:15.596079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:15.596416 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:15.596618 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:15.596857 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 01:25:15.597008 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:25:15.597879 18896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:25:15.664847 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.664373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34396)
0925 01:25:15.664574 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.664665 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:25:15.665978 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:15.665616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43241 (local address 127.0.0.1:34412)
0925 01:25:15.665793 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:15.665866 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 01:25:15.684401 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 18773
I20260925 01:25:15.697834 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 18379
I20260925 01:25:15.699059 18441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:25:15.809002 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 18379
I20260925 01:25:15.831969 7090 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:25:16.166539 7090 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:25:16.166747 7090 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:25:16.501226 7090 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:25:16.501 UTC [17965] LOG: received smart shutdown request
2026-09-25 01:25:16.505 UTC [17965] LOG: background worker "logical replication launcher" (PID 17974) exited with exit code 1
2026-09-25 01:25:16.505 UTC [17969] LOG: shutting down
2026-09-25 01:25:16.663 UTC [17965] LOG: database system is shut down
2026-09-25T01:25:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47848 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:25:16 dist-test-slave-s0b2 krb5kdc[18960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:25:16 dist-test-slave-s0b2 krb5kdc[18960](info): set up 2 sockets
Sep 25 01:25:16 dist-test-slave-s0b2 krb5kdc[18960](info): commencing operation
krb5kdc: starting...
W20260925 01:25:17.787072 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:25:17 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:25:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangeradmin_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangeradmin_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangerlookup_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangerlookup_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
I20260925 01:25:18.071075 7090 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgybJNj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/postgres -l logfile start
2026-09-25 01:25:20.357 UTC [18990] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:25:20.357 UTC [18990] LOG: listening on IPv4 address "127.6.236.148", port 51359
2026-09-25 01:25:20.360 UTC [18990] LOG: listening on Unix socket "/tmp/.s.PGSQL.51359"
2026-09-25 01:25:20.380 UTC [18993] LOG: database system was shut down at 2026-09-25 01:25:19 UTC
2026-09-25 01:25:20.383 UTC [18990] LOG: database system is ready to accept connections
I20260925 01:25:21.278971 7090 mini_postgres.cc:96] Postgres bound to 51359
2026-09-25 01:25:21.294 UTC [19001] FATAL: database "slave" does not exist
127.6.236.148:51359 - accepting connections
I20260925 01:25:21.295444 7090 mini_ranger.cc:161] Starting Ranger...
I20260925 01:25:21.318439 7090 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:25:21.749919 7090 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:25:21.750025 7090 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin
2026-09-25 01:25:22,020 [I] DB FLAVOR :POSTGRES
2026-09-25 01:25:22,020 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:25:22,020 [I] Checking connection..
2026-09-25 01:25:22,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:25:22,272 [I] Checking connection passed.
2026-09-25 01:25:22,272 [I] --------- Verifying version history table ---------
2026-09-25 01:25:22,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:25:22,527 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:25:22,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:25:22,771 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:25:22,771 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:25:22,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:25:23.019 UTC [19065] WARNING: there is no transaction in progress
2026-09-25 01:25:23,025 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:25:23,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:25:23,278 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:25:23,279 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:25:23,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:25:23,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:25:23,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s0b2.c.gcp-upstream.internal','N') ;"
2026-09-25 01:25:24,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:25:24,268 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:25:24,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:25:24,519 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:25:24,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:25:24,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:25:25,017 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:25:25,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:25:25.923 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:25.939 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:25.954 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:25.968 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:25.982 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:25.999 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.049 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.055 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.068 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.078 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.088 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.100 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.109 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.115 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.121 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26.127 UTC [19173] WARNING: there is no transaction in progress
2026-09-25 01:25:26,737 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:25:26,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:25:26,969 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:25:26,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:25:27,222 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:25:27,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:25:27,483 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:25:27,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:25:27,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:25:27,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s0b2.c.gcp-upstream.internal';"
2026-09-25 01:25:28,196 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:25:28,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:25:28,437 [I] DB_PATCHES have already been applied
I20260925 01:25:28.443645 7090 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgybJNj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:25:28.443732 7090 mini_ranger.cc:193] Using host: 127.6.236.148
I20260925 01:25:28.457974 7090 mini_ranger.cc:227] Ranger admin URL: http://127.6.236.148:47009
Sep 25, 2026 1:25:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:25:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:25:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:25:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:25:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:25:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:25:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:25:29.460541 7090 mini_ranger.cc:160] Time spent starting Ranger: real 8.165s user 0.004s sys 0.033s
Sep 25, 2026 1:25:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:25:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 1:25:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:25:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:25:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:25:49.659286 7090 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.236.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.236.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangerkms_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.236.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/rangerkms_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.236.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.236.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.236.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/HTTP_127.6.236.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:25:49 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299549, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:25:49.808589 7090 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:25:49.828444 7090 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:25:50.631588 7090 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:25:50.631686 7090 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms
I20260925 01:25:50.631726 7090 mini_ranger_kms.cc:227] Using postgres at 127.6.236.148:51359
2026-09-25 01:25:50,960 [I] DB FLAVOR :POSTGRES
2026-09-25 01:25:50,960 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:25:50,960 [I] Checking connection
2026-09-25 01:25:50,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:25:51,194 [I] connection success
2026-09-25 01:25:51,194 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:25:51,194 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:25:51,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:25:51,430 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:25:51,430 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:25:51,431 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:25:51,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.236.148:51359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:25:51,669 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:25:52.128780 7090 mini_ranger_kms.cc:310] Created kms service
I20260925 01:25:52.293570 7090 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:25:52.333984 7090 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:25:52 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299549, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:52.699224 7090 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:25:52.699527 7090 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgybJNj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgybJNj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:25:52.699580 7090 mini_ranger_kms.cc:240] Using host: 127.6.236.148
I20260925 01:25:52.717335 7090 mini_ranger_kms.cc:276] Ranger KMS PID: 19353
I20260925 01:25:52.717450 7090 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.236.148:50209
Sep 25, 2026 1:25:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:25:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:25:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:25:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:25:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:25:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 01:25:54.753232 7090 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.945s user 0.061s sys 0.280s
I20260925 01:25:54.753468 7090 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:25:54.753641 7090 mini_ranger_kms.cc:401] 127.6.236.148:50209/kms/v1/keys
Sep 25 01:25:54 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 01:25:55 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:25:55 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299554, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.236.148@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:25:57 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:25:57.946414 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38617
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:38617
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:25:58.255746 19404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:58.256003 19404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:58.256124 19404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:58.263051 19404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:25:58.263150 19404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:58.263221 19404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:58.263286 19404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:25:58.263350 19404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:25:58.272917 19404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.236.148:50209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:38617
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:58.274115 19404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:58.275417 19404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:58.286535 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:58.286568 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:58.286595 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:58.287149 19404 server_base.cc:1034] running on GCE node
I20260925 01:25:58.287900 19404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:58.289204 19404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:58.290386 19404 hybrid_clock.cc:648] HybridClock initialized: now 1790299558290348 us; error 27 us; skew 500 ppm
I20260925 01:25:58.290707 19404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:58 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299558, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:58.293535 19404 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:25:58.294914 19404 webserver.cc:460] Webserver started at http://127.6.236.190:40603/ using document root <none> and password file <none>
I20260925 01:25:58.295514 19404 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:58.295621 19404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:58.295895 19404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:58 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299558, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:58.344470 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e"
format_stamp: "Formatted at 2026-09-25 01:25:58 on dist-test-slave-s0b2"
server_key: "cbd25dff767d8310661acb050b10ddbe"
server_key_iv: "a204e374f12625712e238f652b50891f"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:58.345440 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e"
format_stamp: "Formatted at 2026-09-25 01:25:58 on dist-test-slave-s0b2"
server_key: "cbd25dff767d8310661acb050b10ddbe"
server_key_iv: "a204e374f12625712e238f652b50891f"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:58.351145 19404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:25:58.367262 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:58.368837 19404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:25:58.369022 19404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e"
format_stamp: "Formatted at 2026-09-25 01:25:58 on dist-test-slave-s0b2"
server_key: "cbd25dff767d8310661acb050b10ddbe"
server_key_iv: "a204e374f12625712e238f652b50891f"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:58.369241 19404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:58.384277 19404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:58.388579 19404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:58.388918 19404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:58.406914 19404 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:38617
I20260925 01:25:58.406919 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:38617 every 8 connection(s)
I20260925 01:25:58.408219 19404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:25:58.409561 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 19404
I20260925 01:25:58.409771 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:25:58.410593 7090 external_mini_cluster.cc:1428] Setting key e1f877d55c57a93a4c30e12f213af794
I20260925 01:25:58.413940 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:25:58.429349 19472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Bootstrap starting.
I20260925 01:25:58.432883 19472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Neither blocks nor log segments found. Creating new log.
I20260925 01:25:58.434012 19472 log.cc:824] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:25:58.436882 19472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: No bootstrap required, opened a new log
Sep 25 01:25:58 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:58.447300 19472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:25:58.447794 19472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:25:58.447817 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:58.412721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.6.236.190:38617)
0925 01:25:58.422764 (+ 10043us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:58.422781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:58.423203 (+ 422us) server_negotiation.cc:407] Connection header received
0925 01:25:58.425256 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:58.425287 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:58.425825 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:58.426306 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:58.427872 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:58.429774 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:58.431664 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:58.432665 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:58.432852 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:58.438795 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:58.438806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:58.438827 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:58.438876 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:58.442958 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:58.443841 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:58.443851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:58.443860 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:58.444051 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:58.444454 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:58.444459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:58.444463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:58.445222 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:58.445562 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:58.445972 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 01:25:58.446460 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9795,"thread_start_us":9585,"threads_started":1}
I20260925 01:25:58.447932 19472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf76e1b30d0400e91ffe3f4c1fea88e, State: Initialized, Role: FOLLOWER
I20260925 01:25:58.448647 19472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:25:58.448915 19472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:25:58.449100 19472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:25:58.449323 19472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:25:58.451731 19472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:25:58.452183 19472 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf76e1b30d0400e91ffe3f4c1fea88e; no voters:
I20260925 01:25:58.452771 19472 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:25:58.452876 19477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:25:58.457842 19477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 LEADER]: Becoming Leader. State: Replica: 1cf76e1b30d0400e91ffe3f4c1fea88e, State: Running, Role: LEADER
I20260925 01:25:58.458532 19477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:25:58.460156 19472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:25:58.461309 19479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } } }
I20260925 01:25:58.461709 19479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: This master's current role is: LEADER
I20260925 01:25:58.463881 19478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf76e1b30d0400e91ffe3f4c1fea88e. Latest consensus state: current_term: 1 leader_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } } }
I20260925 01:25:58.464169 19478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: This master's current role is: LEADER
I20260925 01:25:58.465274 19482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:25:58.470593 19482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:25:58.491751 19482 catalog_manager.cc:1347] Generated new cluster ID: 4e03c3a9d8334899b2484001693e45b0
I20260925 01:25:58.491868 19482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:25:58.523792 19482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:25:58.525699 19482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:25:58.534938 19472 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:25:58.564536 19482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Generated new TSK 0
I20260925 01:25:58.565629 19482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:25:59.274392 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:25:59 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:25:59.677847 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:25:59.678308 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:25:59.678457 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:25:59.686144 19517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:25:59.686277 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:25:59.686431 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:25:59.697832 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:25:59.699199 19517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:25:59.700503 19517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:25:59.716843 19527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:59.717301 19528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:25:59.717909 19530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:25:59.718801 19517 server_base.cc:1034] running on GCE node
I20260925 01:25:59.719386 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:25:59.720374 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:25:59.721639 19517 hybrid_clock.cc:648] HybridClock initialized: now 1790299559721581 us; error 53 us; skew 500 ppm
I20260925 01:25:59.721947 19517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:25:59 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:25:59.724992 19517 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:25:59.726447 19517 webserver.cc:460] Webserver started at http://127.6.236.129:45453/ using document root <none> and password file <none>
I20260925 01:25:59.727123 19517 fs_manager.cc:359] Metadata directory not provided
I20260925 01:25:59.727218 19517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:25:59.727497 19517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:25:59 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:25:59.752521 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "c1dc93724f55467b9d24a03442f3dd72"
format_stamp: "Formatted at 2026-09-25 01:25:59 on dist-test-slave-s0b2"
server_key: "88df241654a678aa2b667a4d1da78f2f"
server_key_iv: "7e8637e4114dab3b72afe9834557e1a4"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:59.753362 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "c1dc93724f55467b9d24a03442f3dd72"
format_stamp: "Formatted at 2026-09-25 01:25:59 on dist-test-slave-s0b2"
server_key: "88df241654a678aa2b667a4d1da78f2f"
server_key_iv: "7e8637e4114dab3b72afe9834557e1a4"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:59.758651 19517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:25:59.784749 19537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:59.786285 19517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:25:59.786522 19517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "c1dc93724f55467b9d24a03442f3dd72"
format_stamp: "Formatted at 2026-09-25 01:25:59 on dist-test-slave-s0b2"
server_key: "88df241654a678aa2b667a4d1da78f2f"
server_key_iv: "7e8637e4114dab3b72afe9834557e1a4"
server_key_version: "kuduclusterkey@0"
I20260925 01:25:59.786782 19517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:25:59.805240 19517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:25:59.809389 19517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:25:59.809698 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:25:59.810765 19517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:25:59.812240 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:25:59.812366 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:59.812491 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:25:59.812561 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:25:59.871577 19517 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42637
I20260925 01:25:59.871594 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42637 every 8 connection(s)
I20260925 01:25:59.873179 19517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:25:59.877625 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 19517
I20260925 01:25:59.877854 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:25:59.878762 7090 external_mini_cluster.cc:1428] Setting key a2f50e3c7e8c5280014c5067378da505
Sep 25 01:25:59 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:25:59.901666 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:59.876540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.129:56765)
0925 01:25:59.882865 (+ 6325us) negotiation.cc:107] Waiting for socket to connect
0925 01:25:59.882915 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:25:59.885118 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:25:59.885841 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:25:59.885850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:25:59.886350 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:25:59.886954 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:59.886973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:59.888009 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:59.888013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:59.889280 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:25:59.889292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:25:59.890098 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:25:59.890106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:25:59.890396 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:59.891169 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:25:59.891197 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:25:59.893265 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:25:59.896935 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:59.896945 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:59.896963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:59.897581 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:59.898043 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:25:59.898061 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:25:59.898065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:25:59.898334 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:25:59.898979 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:25:59.898987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:25:59.899416 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 01:25:59.899695 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 01:25:59.900243 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6035,"thread_start_us":5835,"threads_started":1}
I20260925 01:25:59.903596 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:25:59.876107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:56765 (local address 127.6.236.190:38617)
0925 01:25:59.883727 (+ 7620us) server_negotiation.cc:206] Beginning negotiation
0925 01:25:59.883733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:25:59.883756 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:25:59.885384 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:25:59.885391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:25:59.885475 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:25:59.885579 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:25:59.887117 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:59.887861 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:59.889417 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:25:59.889869 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:25:59.889988 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:25:59.893484 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:25:59.893489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:25:59.893493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:25:59.893524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:25:59.896739 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:59.897747 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:59.897753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:59.897757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:59.897907 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:25:59.898487 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:25:59.898491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:25:59.898495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:25:59.898831 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:25:59.902826 (+ 3995us) server_negotiation.cc:1035] Waiting for connection context
0925 01:25:59.902973 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 01:25:59.903225 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7492,"thread_start_us":7365,"threads_started":1}
I20260925 01:25:59.905231 19650 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:25:59.905560 19650 heartbeater.cc:464] Registering TS with master...
I20260925 01:25:59.906469 19650 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:25:59.909755 19437 ts_manager.cc:194] Registered new tserver with Master: c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129:42637)
I20260925 01:25:59.911823 19437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:56765
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:25:59.968969 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:26:00 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
Sep 25 01:26:00 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
W20260925 01:26:00.368351 19656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:00.368731 19656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:00.368924 19656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:00.378412 19656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:00.378556 19656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:00.378751 19656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:00.390959 19656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:00.392724 19656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:00.394420 19656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:00.418438 19663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:00.418346 19664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:00.420955 19666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:00.421228 19656 server_base.cc:1034] running on GCE node
I20260925 01:26:00.421978 19656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:00.423213 19656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:00.424445 19656 hybrid_clock.cc:648] HybridClock initialized: now 1790299560424373 us; error 56 us; skew 500 ppm
I20260925 01:26:00.424849 19656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:00 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:00.428452 19656 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:00.429942 19656 webserver.cc:460] Webserver started at http://127.6.236.130:45571/ using document root <none> and password file <none>
I20260925 01:26:00.430691 19656 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:00.430814 19656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:00.431097 19656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:26:00 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:00.457545 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "ef4ebed07fef4631babc0a28696ca191"
format_stamp: "Formatted at 2026-09-25 01:26:00 on dist-test-slave-s0b2"
server_key: "23a44b126bbd44832caf1d0e904c0fc8"
server_key_iv: "2f10cfce84a16d4221a4b3672403140d"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:00.458709 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "ef4ebed07fef4631babc0a28696ca191"
format_stamp: "Formatted at 2026-09-25 01:26:00 on dist-test-slave-s0b2"
server_key: "23a44b126bbd44832caf1d0e904c0fc8"
server_key_iv: "2f10cfce84a16d4221a4b3672403140d"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:00.472774 19656 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260925 01:26:00.496560 19673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:00.498713 19656 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.005s
I20260925 01:26:00.498914 19656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "ef4ebed07fef4631babc0a28696ca191"
format_stamp: "Formatted at 2026-09-25 01:26:00 on dist-test-slave-s0b2"
server_key: "23a44b126bbd44832caf1d0e904c0fc8"
server_key_iv: "2f10cfce84a16d4221a4b3672403140d"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:00.499311 19656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:00.564980 19656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:00.570458 19656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:00.570863 19656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:00.578337 19656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:00.586691 19656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:00.586863 19656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:00.587042 19656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:00.587168 19656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:00.776067 19656 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:46179
I20260925 01:26:00.776113 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:46179 every 8 connection(s)
I20260925 01:26:00.777671 19656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:00.782018 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 19656
I20260925 01:26:00.782176 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:26:00.782678 7090 external_mini_cluster.cc:1428] Setting key 098e613841976ea906853724ba6625e2
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Sep 25 01:26:00 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:00.827257 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:00.781823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:56709 (local address 127.6.236.190:38617)
0925 01:26:00.804505 (+ 22682us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:00.804514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:00.805534 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 01:26:00.808852 (+ 3318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:00.808859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:00.808955 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:00.809032 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:00.811323 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:00.812441 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:00.814870 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:00.815509 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:00.815601 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:00.820064 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:00.820071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:00.820076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:00.820118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:00.823533 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:00.824437 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:00.824443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:00.824447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:00.824639 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:00.825159 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:00.825165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:00.825168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:00.825527 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:00.825652 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:00.826625 (+ 973us) server_negotiation.cc:299] Negotiation successful
0925 01:26:00.826865 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22398,"thread_start_us":22576,"threads_started":1}
I20260925 01:26:00.829923 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:00.781236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.130:56709)
0925 01:26:00.805096 (+ 23860us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:00.805159 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:00.807603 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:00.809595 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:00.809607 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:00.810308 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:00.811116 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:00.811140 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:00.812640 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:00.812645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:00.814690 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:00.814705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:00.815950 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:00.815956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:00.816310 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:00.817069 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:00.817100 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:00.819795 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:00.823710 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:00.823719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:00.823738 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:00.824309 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:00.824775 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:00.824780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:00.824784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:00.825038 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:00.825747 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:00.825755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:00.826177 (+ 422us) client_negotiation.cc:769] Sending connection context
0925 01:26:00.827803 (+ 1626us) client_negotiation.cc:240] Negotiation successful
0925 01:26:00.828448 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23521,"thread_start_us":26594,"threads_started":1}
I20260925 01:26:00.831935 19793 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:26:00.832253 19793 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:00.833047 19793 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:00.834784 19437 ts_manager.cc:194] Registered new tserver with Master: ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:00.835881 19437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:56709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:26:00.847254 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:00.915555 19650 heartbeater.cc:503] Master 127.6.236.190:38617 was elected leader, sending a full tablet report...
W20260925 01:26:01.215492 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:01.215792 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:01.215925 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:01.222869 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:01.222962 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:01.223110 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:26:01.232283 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:01.233443 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:01.234691 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:01.246459 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:01.246539 19807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:01.246490 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:01.247283 19799 server_base.cc:1034] running on GCE node
I20260925 01:26:01.247865 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:01.248739 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:01.249940 19799 hybrid_clock.cc:648] HybridClock initialized: now 1790299561249879 us; error 54 us; skew 500 ppm
I20260925 01:26:01.250222 19799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:01.253213 19799 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:01.254585 19799 webserver.cc:460] Webserver started at http://127.6.236.131:44279/ using document root <none> and password file <none>
I20260925 01:26:01.255211 19799 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:01.255316 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:01.255590 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:01.272946 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "227fa41aa4a44b6daea779e42c4d03cc"
format_stamp: "Formatted at 2026-09-25 01:26:01 on dist-test-slave-s0b2"
server_key: "eaf5bf4f1a5fbb27dfb7d68fc1bc3426"
server_key_iv: "eb2c2bb1dbffb03e3baf8ca6c67d7360"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:01.274050 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "227fa41aa4a44b6daea779e42c4d03cc"
format_stamp: "Formatted at 2026-09-25 01:26:01 on dist-test-slave-s0b2"
server_key: "eaf5bf4f1a5fbb27dfb7d68fc1bc3426"
server_key_iv: "eb2c2bb1dbffb03e3baf8ca6c67d7360"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:01.280800 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 01:26:01.295593 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:01.297047 19799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:01.297214 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "227fa41aa4a44b6daea779e42c4d03cc"
format_stamp: "Formatted at 2026-09-25 01:26:01 on dist-test-slave-s0b2"
server_key: "eaf5bf4f1a5fbb27dfb7d68fc1bc3426"
server_key_iv: "eb2c2bb1dbffb03e3baf8ca6c67d7360"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:01.297425 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:01.312664 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:01.316615 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:01.316886 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:01.317826 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:01.319176 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:01.319257 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:01.319370 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:01.319423 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:01.348986 19799 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:43393
I20260925 01:26:01.349009 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:43393 every 8 connection(s)
I20260925 01:26:01.350569 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:01.360870 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 19799
I20260925 01:26:01.361034 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:26:01.361585 7090 external_mini_cluster.cc:1428] Setting key c0df95653075910df59dfca5eb961e0c
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:01.369107 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:01.352900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:60077 (local address 127.6.236.190:38617)
0925 01:26:01.353376 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:01.353381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:01.354186 (+ 805us) server_negotiation.cc:407] Connection header received
0925 01:26:01.355930 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:01.355937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:01.356019 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:01.356115 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:01.357547 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.358325 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.359862 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.360279 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.360388 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:01.363758 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:01.363765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:01.363769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:01.363806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:01.366483 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.367143 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.367147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.367150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.367291 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.367663 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.367668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.367670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.367949 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:01.368073 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:01.368695 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 01:26:01.368864 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":224,"threads_started":1}
I20260925 01:26:01.369939 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:01.353375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.131:60077)
0925 01:26:01.354038 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:01.354075 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:01.355664 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:01.356396 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:01.356405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:01.356806 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:01.357379 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:01.357398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.358471 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.358475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:01.359706 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:01.359718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.360517 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.360525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:01.360741 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:01.361398 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:01.361420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:01.363528 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:01.366587 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:01.366594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:01.366606 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:01.367022 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:01.367382 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:01.367386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:01.367388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:01.367554 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:01.368048 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:01.368053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:01.368350 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 01:26:01.368563 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 01:26:01.368921 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":268,"threads_started":1}
I20260925 01:26:01.371174 19927 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:26:01.371429 19927 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:01.372092 19927 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:01.373355 19437 ts_manager.cc:194] Registered new tserver with Master: 227fa41aa4a44b6daea779e42c4d03cc (127.6.236.131:43393)
I20260925 01:26:01.374182 19437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:60077
I20260925 01:26:01.376789 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:26:01.378556 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 19517
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:26:01.394080 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:01.379479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.6.236.129:42637)
0925 01:26:01.380050 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:01.380056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:01.380080 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:26:01.380133 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:01.380139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:01.380318 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:01.380438 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:01.381709 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.382781 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.384173 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.384680 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.384771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:01.388778 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:01.388790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:01.388804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:01.388842 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:01.391640 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.392307 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.392314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.392316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.392472 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.392780 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.392783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.392785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.393127 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:01.393354 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:01.393558 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 01:26:01.393790 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260925 01:26:01.395316 19645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:01.477404 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 19517
I20260925 01:26:01.490901 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 19656
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:26:01.506345 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:01.491891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.6.236.130:46179)
0925 01:26:01.492413 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:01.492420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:01.492445 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:26:01.492503 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:01.492509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:01.492721 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:01.492840 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:01.493994 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.495050 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.496351 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.496872 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.496960 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:01.501057 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:01.501068 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:01.501082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:01.501139 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:01.503791 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.504450 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.504457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.504459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.504582 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.504883 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.504886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.504888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.505235 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:01.505455 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:01.505709 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 01:26:01.505910 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260925 01:26:01.507539 19788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:01.587343 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 19656
I20260925 01:26:01.600436 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 19799
Sep 25 01:26:01 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:26:01.616396 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:01.601459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.6.236.131:43393)
0925 01:26:01.602151 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:01.602158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:01.602189 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:26:01.602282 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:01.602291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:01.602517 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:01.602663 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:01.603906 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.604937 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.606130 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:01.606764 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:01.606860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:01.610992 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:01.611003 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:01.611017 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:01.611055 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:01.613511 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.614262 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.614272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.614276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.614421 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:01.614812 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:01.614817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:01.614819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:01.615168 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:01.615475 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:01.615789 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 01:26:01.616057 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":397,"threads_started":1}
I20260925 01:26:01.617719 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:01.699281 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 19799
I20260925 01:26:01.712355 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 19404
I20260925 01:26:01.713567 19467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:01.817005 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 19404
I20260925 01:26:01.837929 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38617
--webserver_interface=127.6.236.190
--webserver_port=40603
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:38617
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:02.143563 19939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:02.143826 19939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:02.143937 19939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:02.151095 19939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:02.151196 19939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:02.151268 19939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:02.151341 19939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:02.151403 19939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:02.161310 19939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:38617
--ranger_config_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=40603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:02.162592 19939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:02.163878 19939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:02.174873 19944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:02.174912 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:02.174952 19945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:02.175926 19939 server_base.cc:1034] running on GCE node
I20260925 01:26:02.176574 19939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:02.177855 19939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:02.179036 19939 hybrid_clock.cc:648] HybridClock initialized: now 1790299562178980 us; error 56 us; skew 500 ppm
I20260925 01:26:02.179337 19939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:02 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:02.182315 19939 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:02.183596 19939 webserver.cc:460] Webserver started at http://127.6.236.190:40603/ using document root <none> and password file <none>
I20260925 01:26:02.184186 19939 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:02.184293 19939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:26:02 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:02.253590 19939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:26:02.256788 19955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:02.258384 19939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:02.258553 19939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e"
format_stamp: "Formatted at 2026-09-25 01:25:58 on dist-test-slave-s0b2"
server_key: "cbd25dff767d8310661acb050b10ddbe"
server_key_iv: "a204e374f12625712e238f652b50891f"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:02.259531 19939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:02.293296 19939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:02.297288 19939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:02.297552 19939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:02.314666 19939 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:38617
I20260925 01:26:02.314697 20006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:38617 every 8 connection(s)
I20260925 01:26:02.316063 19939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:02.319056 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 19939
I20260925 01:26:02.320039 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:42637
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:38617
--webserver_port=45453
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:02.324133 20007 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:26:02.328347 20007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Bootstrap starting.
I20260925 01:26:02.346222 20007 log.cc:824] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:02.354707 20007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:26:02.355157 20007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Bootstrap complete.
I20260925 01:26:02.363888 20007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:26:02.364754 20007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf76e1b30d0400e91ffe3f4c1fea88e, State: Initialized, Role: FOLLOWER
I20260925 01:26:02.365311 20007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:26:02.365509 20007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:02.365612 20007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:02.365759 20007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:26:02.369469 20007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:26:02.369944 20007 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf76e1b30d0400e91ffe3f4c1fea88e; no voters:
I20260925 01:26:02.370512 20007 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:26:02.370647 20011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:26:02.371714 20011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [term 2 LEADER]: Becoming Leader. State: Replica: 1cf76e1b30d0400e91ffe3f4c1fea88e, State: Running, Role: LEADER
I20260925 01:26:02.372300 20011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } }
I20260925 01:26:02.372601 20007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:02.374722 20013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf76e1b30d0400e91ffe3f4c1fea88e. Latest consensus state: current_term: 2 leader_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } } }
I20260925 01:26:02.375088 20013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:02.375603 20012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf76e1b30d0400e91ffe3f4c1fea88e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38617 } } }
I20260925 01:26:02.375861 20012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:02.386379 20016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:02.392257 20016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:02.393927 20016 catalog_manager.cc:1259] Loaded cluster ID: 4e03c3a9d8334899b2484001693e45b0
I20260925 01:26:02.394074 20016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:02.397890 20016 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:02.399380 20016 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Loaded TSK: 0
I20260925 01:26:02.400960 20016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:26:02.411232 20007 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 01:26:02.723146 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:02.723554 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:02.723752 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:02.734877 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:02.735083 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:02.735307 20009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:26:02.750412 20009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:42637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=45453
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:02.752287 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:02.754015 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:02.782047 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:02.786604 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:02.790323 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:02.791069 20009 server_base.cc:1034] running on GCE node
I20260925 01:26:02.792052 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:02.798118 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:02.799464 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790299562799387 us; error 71 us; skew 500 ppm
I20260925 01:26:02.799885 20009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:02 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:02.810976 20009 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:26:02.816120 20009 webserver.cc:460] Webserver started at http://127.6.236.129:45453/ using document root <none> and password file <none>
I20260925 01:26:02.817216 20009 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:02.817418 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:26:02 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:02.858227 20009 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260925 01:26:02.868388 20055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:02.870318 20009 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.001s
I20260925 01:26:02.870546 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "c1dc93724f55467b9d24a03442f3dd72"
format_stamp: "Formatted at 2026-09-25 01:25:59 on dist-test-slave-s0b2"
server_key: "88df241654a678aa2b667a4d1da78f2f"
server_key_iv: "7e8637e4114dab3b72afe9834557e1a4"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:02.872030 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:02.914214 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:02.919293 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:02.919682 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:02.926961 20009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:02.934895 20009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:02.935084 20009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:02.935257 20009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:02.935377 20009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:02.984894 20009 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42637
I20260925 01:26:02.984936 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42637 every 8 connection(s)
I20260925 01:26:02.986461 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:26:02.992662 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20009
I20260925 01:26:02.993536 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:46179
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:38617
--webserver_port=45571
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:26:03 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:03.020879 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:02.991260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.129:56275)
0925 01:26:02.995754 (+ 4494us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:02.995812 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:02.998044 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:03.000907 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:03.000918 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:03.001514 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:03.002306 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:03.002338 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.004796 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.004802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:03.006682 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:03.006695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.007737 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.007743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:03.008006 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:03.008778 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:03.008805 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:03.011465 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:03.015733 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:03.015743 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:03.015760 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:03.016359 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:03.016917 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:03.016922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:03.016926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:03.017173 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:03.018167 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:03.018175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:03.018612 (+ 437us) client_negotiation.cc:769] Sending connection context
0925 01:26:03.018897 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 01:26:03.019451 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4181,"thread_start_us":3966,"threads_started":1}
I20260925 01:26:03.024039 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:02.991195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:56275 (local address 127.6.236.190:38617)
0925 01:26:02.998872 (+ 7677us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:02.998891 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:02.998944 (+ 53us) server_negotiation.cc:407] Connection header received
0925 01:26:02.999384 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:02.999404 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:02.999867 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:03.000257 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:03.002557 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.004589 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.006820 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.007505 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.007712 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:03.011791 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:03.011801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:03.011820 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:03.011869 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:03.015564 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:03.016549 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:03.016559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:03.016567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:03.016778 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:03.017326 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:03.017331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:03.017335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:03.017985 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:03.022109 (+ 4124us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:03.022394 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 01:26:03.022857 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7438,"thread_start_us":7194,"threads_started":1}
I20260925 01:26:03.027544 20170 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:26:03.027830 20170 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:03.028627 20170 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
W20260925 01:26:03.031711 20170 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:38617 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:26:03.031912 20170 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:03.032155 20170 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:03.033380 20170 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:03.033622 20170 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
W20260925 01:26:03.034668 20170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:26:03.364197 20174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:03.364472 20174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:03.364645 20174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:03.372351 20174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:03.372485 20174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:03.372730 20174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:03.382699 20174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:46179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=45571
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:03.383958 20174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:03.385308 20174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:03.401182 20187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:03.401207 20186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:03.402084 20190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:03.404496 20174 server_base.cc:1034] running on GCE node
I20260925 01:26:03.405256 20174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:03.406479 20174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:03.407766 20174 hybrid_clock.cc:648] HybridClock initialized: now 1790299563407708 us; error 71 us; skew 500 ppm
I20260925 01:26:03.408159 20174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:03 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:03.412835 20174 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:03.415699 20174 webserver.cc:460] Webserver started at http://127.6.236.130:45571/ using document root <none> and password file <none>
I20260925 01:26:03.416648 20174 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:03.416851 20174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:26:03 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:03.445441 20174 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:26:03.453652 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:03.455592 20174 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260925 01:26:03.455844 20174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "ef4ebed07fef4631babc0a28696ca191"
format_stamp: "Formatted at 2026-09-25 01:26:00 on dist-test-slave-s0b2"
server_key: "23a44b126bbd44832caf1d0e904c0fc8"
server_key_iv: "2f10cfce84a16d4221a4b3672403140d"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:03.457286 20174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:03.485153 20174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:03.490312 20174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:03.490648 20174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:03.501309 20174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:03.503109 20174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:03.503252 20174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:26:03.503378 20174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:03.503470 20174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:26:03 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:03.598582 20174 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:46179
I20260925 01:26:03.600571 20174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:03.602614 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:46179 every 8 connection(s)
I20260925 01:26:03.606173 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20174
I20260925 01:26:03.607020 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.236.148:50209/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:43393
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:38617
--webserver_port=44279
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:26:03 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:03.639194 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:03.610536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:44079 (local address 127.6.236.190:38617)
0925 01:26:03.624250 (+ 13714us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:03.624256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:03.624275 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:26:03.624335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:03.624339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:03.624410 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:03.624510 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:03.626663 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.628114 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.629942 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.630407 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.630518 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:03.633810 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:03.633815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:03.633818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:03.633852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:03.636314 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:03.636974 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:03.636980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:03.636984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:03.637112 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:03.637514 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:03.637519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:03.637522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:03.637792 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:03.637931 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:03.638626 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 01:26:03.638836 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13579,"thread_start_us":13454,"threads_started":1}
I20260925 01:26:03.640193 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:03.610163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.130:44079)
0925 01:26:03.620812 (+ 10649us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:03.620870 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:03.623160 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:03.624816 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:03.624828 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:03.625497 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:03.626428 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:03.626456 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.628488 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.628494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:03.629777 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:03.629790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:03.630638 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:03.630646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:03.630859 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:03.631570 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:03.631611 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:03.633600 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:03.636442 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:03.636451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:03.636467 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:03.636865 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:03.637214 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:03.637219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:03.637223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:03.637421 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:03.637915 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:03.637922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:03.638267 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 01:26:03.638536 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 01:26:03.639047 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10316,"thread_start_us":9587,"threads_started":1}
I20260925 01:26:03.641708 20314 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:26:03.642012 20314 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:03.642725 20314 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:03.644505 19972 ts_manager.cc:194] Registered new tserver with Master: ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:03.646013 19972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:44079
W20260925 01:26:04.002574 20317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:04.003057 20317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:04.003291 20317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:04.014705 20317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:04.014850 20317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:04.015077 20317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
Sep 25 01:26:04 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:04.030396 20317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.236.148:50209/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:43393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=44279
--tserver_master_addrs=127.6.236.190:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:04.032181 20317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:04.033847 20317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:26:04.034811 20170 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:04.035218 20170 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:04.036387 19972 ts_manager.cc:194] Registered new tserver with Master: c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129:42637)
I20260925 01:26:04.037544 19972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:56275
W20260925 01:26:04.094035 20325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:04.101531 20324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:04.103116 20317 server_base.cc:1034] running on GCE node
W20260925 01:26:04.103788 20327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:04.104650 20317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:04.110913 20317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:04.112406 20317 hybrid_clock.cc:648] HybridClock initialized: now 1790299564112297 us; error 103 us; skew 500 ppm
I20260925 01:26:04.112881 20317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:04 dist-test-slave-s0b2 krb5kdc[18960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299564, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:04.121275 20317 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:04.160458 20317 webserver.cc:460] Webserver started at http://127.6.236.131:44279/ using document root <none> and password file <none>
I20260925 01:26:04.161582 20317 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:04.161896 20317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:26:04 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299564, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:04.210002 20317 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260925 01:26:04.219821 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:04.221710 20317 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.000s
I20260925 01:26:04.221946 20317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "227fa41aa4a44b6daea779e42c4d03cc"
format_stamp: "Formatted at 2026-09-25 01:26:01 on dist-test-slave-s0b2"
server_key: "eaf5bf4f1a5fbb27dfb7d68fc1bc3426"
server_key_iv: "eb2c2bb1dbffb03e3baf8ca6c67d7360"
server_key_version: "kuduclusterkey@0"
I20260925 01:26:04.223450 20317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:04.291045 20317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:04.296021 20317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:04.296324 20317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:04.309502 20317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:04.311487 20317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:04.311674 20317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:26:04.311797 20317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:04.311931 20317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:26:04 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for HTTP/127.6.236.148@KRBTEST.COM
I20260925 01:26:04.522574 20317 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:43393
I20260925 01:26:04.522604 20448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:43393 every 8 connection(s)
I20260925 01:26:04.524479 20317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:04.531410 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20317
Sep 25 01:26:04 dist-test-slave-s0b2 krb5kdc[18960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299564, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:04.554085 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.532771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.6.236.190:38617)
0925 01:26:04.533263 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.533269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.537696 (+ 4427us) server_negotiation.cc:407] Connection header received
0925 01:26:04.537888 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.537895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.538001 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.538115 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:04.541312 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.542486 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.544369 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.544935 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.545018 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.548261 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:04.548268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:04.548273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:04.548308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:04.551135 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.551900 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.551905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.551909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.552076 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.552509 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.552515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.552518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.552862 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:04.553134 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.553494 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.553680 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":366,"spinlock_wait_cycles":3328,"thread_start_us":319,"threads_started":1}
I20260925 01:26:04.558758 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.527647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:32993 (local address 127.6.236.190:38617)
0925 01:26:04.532524 (+ 4877us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.532531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.538578 (+ 6047us) server_negotiation.cc:407] Connection header received
0925 01:26:04.541152 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.541157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.541234 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.541327 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:04.543514 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.544662 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.546965 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.547379 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.547483 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.551959 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:04.551965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:04.551969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:04.552013 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:04.555234 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.556322 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.556328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.556330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.556454 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.557165 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.557169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.557171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.557413 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:04.557515 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.558359 (+ 844us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.558497 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":4740,"thread_start_us":4954,"threads_started":1}
I20260925 01:26:04.560210 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.528408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38617 (local address 127.6.236.131:32993)
0925 01:26:04.538405 (+ 9997us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.538463 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.540845 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.541762 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.541775 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.542395 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:04.543256 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.543296 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.545215 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.545219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.546832 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.546844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.547593 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.547601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.547812 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.548956 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:04.548988 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:04.551708 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:04.555353 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:04.555363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:04.555381 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:04.556188 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:04.556566 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:04.556572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:04.556575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:04.557061 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:04.557535 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:04.557544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:04.557962 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.558298 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.558824 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9662,"thread_start_us":9429,"threads_started":1}
I20260925 01:26:04.561533 20451 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38617
I20260925 01:26:04.561888 20451 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:04.562747 20451 heartbeater.cc:511] Master 127.6.236.190:38617 requested a full tablet report, sending...
I20260925 01:26:04.564509 19972 ts_manager.cc:194] Registered new tserver with Master: 227fa41aa4a44b6daea779e42c4d03cc (127.6.236.131:43393)
I20260925 01:26:04.565302 19972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:32993
I20260925 01:26:04.569131 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:26:04.584844 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.572017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.6.236.190:38617)
0925 01:26:04.572180 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.572184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.572714 (+ 530us) server_negotiation.cc:407] Connection header received
0925 01:26:04.572897 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.572903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.572968 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.573043 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:04.574678 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.575369 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.576706 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.577130 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.577235 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.579825 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:04.579831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:04.579834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:04.579865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:04.582314 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.582945 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.582951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.582955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.583081 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:04.583417 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:04.583422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:04.583426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:04.583693 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:04.583912 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.584225 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.584538 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:26:04.589556 19972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:26:04.596768 19972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:26:04.623322 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.610725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.6.236.129:42637)
0925 01:26:04.612530 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.612540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.613213 (+ 673us) server_negotiation.cc:407] Connection header received
0925 01:26:04.614669 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.614678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.614928 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.615091 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.616051 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.617818 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.620424 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.621869 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.621998 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.622220 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.622854 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.623006 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":4136,"threads_started":1}
I20260925 01:26:04.623335 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.610627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42637 (local address 127.0.0.1:43174)
0925 01:26:04.613057 (+ 2430us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.613097 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.613373 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.615408 (+ 2035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.615415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.615445 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.615834 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.615848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.618060 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.618066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.620288 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.620298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.622399 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.622410 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.622604 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.622674 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.622785 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.622963 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2180,"thread_start_us":2058,"threads_started":1}
I20260925 01:26:04.625043 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.612186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:46179 (local address 127.0.0.1:34570)
0925 01:26:04.616439 (+ 4253us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.616457 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.616563 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.618156 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.618161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.618193 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.618728 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.618740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.620493 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.620498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.622531 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.622544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.624334 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.624340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.624502 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.624533 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.624614 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.624737 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4140,"thread_start_us":6330,"threads_started":1}
I20260925 01:26:04.625245 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.612702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.6.236.130:46179)
0925 01:26:04.617584 (+ 4882us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.617592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.617617 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:26:04.617708 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.617715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.617957 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.618145 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.618908 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.620302 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.622712 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.624130 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.624261 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.624468 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.624763 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.624923 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4730,"thread_start_us":4590,"threads_started":1}
I20260925 01:26:04.626402 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.614263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:43393 (local address 127.0.0.1:47394)
0925 01:26:04.618847 (+ 4584us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.618864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.618984 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.619542 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.619547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.619572 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.619939 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.619953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.621395 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.621399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.623380 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.623395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.625617 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.625624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.625792 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.625823 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.625914 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.626035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4454,"mutex_wait_us":1,"thread_start_us":1787,"threads_started":1}
I20260925 01:26:04.626493 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.613911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.6.236.131:43393)
0925 01:26:04.617079 (+ 3168us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.617087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.618952 (+ 1865us) server_negotiation.cc:407] Connection header received
0925 01:26:04.619120 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.619128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.619351 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.619535 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.620140 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.621229 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.623552 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.625095 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.625201 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.625408 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.625979 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.626130 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2994,"thread_start_us":2866,"threads_started":1}
I20260925 01:26:04.627681 20105 tablet_service.cc:1467] Processing CreateTablet for tablet cb28515795d0405bbe24a9687f953447 (DEFAULT_TABLE table=test-table [id=22202823225e441a9a8eaa9573979044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:04.628962 20105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb28515795d0405bbe24a9687f953447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:04.628935 20249 tablet_service.cc:1467] Processing CreateTablet for tablet cb28515795d0405bbe24a9687f953447 (DEFAULT_TABLE table=test-table [id=22202823225e441a9a8eaa9573979044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:04.630041 20384 tablet_service.cc:1467] Processing CreateTablet for tablet cb28515795d0405bbe24a9687f953447 (DEFAULT_TABLE table=test-table [id=22202823225e441a9a8eaa9573979044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:04.630746 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb28515795d0405bbe24a9687f953447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:04.631410 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb28515795d0405bbe24a9687f953447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:04.640635 20477 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Bootstrap starting.
I20260925 01:26:04.644146 20477 tablet_bootstrap.cc:654] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:04.645212 20477 log.cc:824] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:04.645630 20479 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Bootstrap starting.
I20260925 01:26:04.647580 20478 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Bootstrap starting.
I20260925 01:26:04.649289 20479 tablet_bootstrap.cc:654] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:04.649546 20314 heartbeater.cc:503] Master 127.6.236.190:38617 was elected leader, sending a full tablet report...
I20260925 01:26:04.650625 20479 log.cc:824] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:04.652494 20478 tablet_bootstrap.cc:654] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:04.652998 20477 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: No bootstrap required, opened a new log
I20260925 01:26:04.653271 20477 ts_tablet_manager.cc:1397] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260925 01:26:04.653717 20478 log.cc:824] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:04.655922 20479 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: No bootstrap required, opened a new log
I20260925 01:26:04.656176 20479 ts_tablet_manager.cc:1397] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 01:26:04.657790 20478 tablet_bootstrap.cc:492] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: No bootstrap required, opened a new log
I20260925 01:26:04.658085 20478 ts_tablet_manager.cc:1397] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:26:04.663429 20477 raft_consensus.cc:348] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.663851 20477 raft_consensus.cc:374] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:04.663947 20477 raft_consensus.cc:729] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dc93724f55467b9d24a03442f3dd72, State: Initialized, Role: FOLLOWER
I20260925 01:26:04.664553 20477 consensus_queue.cc:260] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.665625 20170 heartbeater.cc:503] Master 127.6.236.190:38617 was elected leader, sending a full tablet report...
I20260925 01:26:04.666074 20478 raft_consensus.cc:348] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.666363 20477 ts_tablet_manager.cc:1428] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260925 01:26:04.666111 20479 raft_consensus.cc:348] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.666471 20478 raft_consensus.cc:374] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:04.666531 20479 raft_consensus.cc:374] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:04.666553 20478 raft_consensus.cc:729] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4ebed07fef4631babc0a28696ca191, State: Initialized, Role: FOLLOWER
I20260925 01:26:04.666618 20479 raft_consensus.cc:729] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227fa41aa4a44b6daea779e42c4d03cc, State: Initialized, Role: FOLLOWER
I20260925 01:26:04.667089 20478 consensus_queue.cc:260] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.667265 20479 consensus_queue.cc:260] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.668721 20478 ts_tablet_manager.cc:1428] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260925 01:26:04.669219 20451 heartbeater.cc:503] Master 127.6.236.190:38617 was elected leader, sending a full tablet report...
I20260925 01:26:04.669761 20479 ts_tablet_manager.cc:1428] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 01:26:04.746754 20171 tablet.cc:2369] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:26:04.782159 20452 tablet.cc:2369] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:26:04.880008 20318 tablet.cc:2369] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:04.889467 20483 raft_consensus.cc:482] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:04.889771 20483 raft_consensus.cc:504] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.891479 20483 leader_election.cc:288] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 227fa41aa4a44b6daea779e42c4d03cc (127.6.236.131:43393), ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:04.893347 20485 raft_consensus.cc:482] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:04.893662 20485 raft_consensus.cc:504] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.899274 20485 leader_election.cc:288] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129:42637), ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:04.899852 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.891879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:43393 (local address 127.6.236.129:45265)
0925 01:26:04.892646 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.892664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.892781 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.893057 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.893061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.893084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.893419 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.893429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.894916 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.894921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.897161 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.897174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.899043 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.899048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.899233 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.899268 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.899355 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.899475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":36,"thread_start_us":544,"threads_started":1}
I20260925 01:26:04.900240 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.891926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:45265 (local address 127.6.236.131:43393)
0925 01:26:04.892253 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.892259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.892760 (+ 501us) server_negotiation.cc:407] Connection header received
0925 01:26:04.892899 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.892903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.892959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.893033 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.893623 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.894780 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.897298 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.898615 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.898752 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.898816 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.899552 (+ 736us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.899718 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260925 01:26:04.900607 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.892087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:33851 (local address 127.6.236.130:46179)
0925 01:26:04.892430 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.892435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.892666 (+ 231us) server_negotiation.cc:407] Connection header received
0925 01:26:04.892872 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.892877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.892951 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.893022 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.893809 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.894830 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.897350 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.898595 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.898726 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.898805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.900219 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.900384 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260925 01:26:04.901264 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.891770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:46179 (local address 127.6.236.129:33851)
0925 01:26:04.892543 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.892561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.892722 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.893258 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.893263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.893292 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.893663 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.893674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.894998 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.895002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.897172 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.897187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.899014 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.899024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.900017 (+ 993us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.900050 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.900749 (+ 699us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.900874 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":9,"thread_start_us":360,"threads_started":1}
I20260925 01:26:04.901670 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "c1dc93724f55467b9d24a03442f3dd72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227fa41aa4a44b6daea779e42c4d03cc" is_pre_election: true
I20260925 01:26:04.902170 20404 raft_consensus.cc:2454] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1dc93724f55467b9d24a03442f3dd72 in term 0.
I20260925 01:26:04.902328 20269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "c1dc93724f55467b9d24a03442f3dd72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4ebed07fef4631babc0a28696ca191" is_pre_election: true
I20260925 01:26:04.902745 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.895640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42637 (local address 127.6.236.131:55473)
0925 01:26:04.895878 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.895895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.895992 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.896731 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.896736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.896763 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.896952 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.896963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.898167 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.898171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.900136 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.900149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.901870 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.901876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.902147 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.902183 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.902343 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.902483 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2}
I20260925 01:26:04.902814 20269 raft_consensus.cc:2454] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1dc93724f55467b9d24a03442f3dd72 in term 0.
I20260925 01:26:04.902988 20056 leader_election.cc:302] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227fa41aa4a44b6daea779e42c4d03cc, c1dc93724f55467b9d24a03442f3dd72; no voters:
I20260925 01:26:04.903234 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.896040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:55473 (local address 127.6.236.129:42637)
0925 01:26:04.896217 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.896223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.896239 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:26:04.896286 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.896291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.896367 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.896441 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.897108 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.898047 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.900340 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.901528 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.901676 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.901740 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.902893 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.903020 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 01:26:04.903466 20483 raft_consensus.cc:2781] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:04.903690 20483 raft_consensus.cc:482] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:04.903805 20483 raft_consensus.cc:3037] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:04.904220 20125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "227fa41aa4a44b6daea779e42c4d03cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1dc93724f55467b9d24a03442f3dd72" is_pre_election: true
I20260925 01:26:04.906131 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.895651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:46179 (local address 127.6.236.131:50485)
0925 01:26:04.897685 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:04.897706 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:04.897804 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:04.899273 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:04.899278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:04.899302 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:04.899701 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.899711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.901303 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.901309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.903503 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:04.903516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.905564 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.905571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:04.905685 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.905725 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:26:04.905807 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:26:04.905892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"mutex_wait_us":861,"thread_start_us":1589,"threads_started":1}
I20260925 01:26:04.906311 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.896195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50485 (local address 127.6.236.130:46179)
0925 01:26:04.898794 (+ 2599us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.898800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.898823 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:26:04.898877 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.898882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.898945 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.899036 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:04.899899 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.901132 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.903683 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.905225 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.905378 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.905425 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.905852 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.905981 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2455,"thread_start_us":2627,"threads_started":1}
I20260925 01:26:04.906648 20483 raft_consensus.cc:504] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.907060 20125 raft_consensus.cc:2379] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 227fa41aa4a44b6daea779e42c4d03cc in current term 1: Already voted for candidate c1dc93724f55467b9d24a03442f3dd72 in this term.
I20260925 01:26:04.907235 20269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "227fa41aa4a44b6daea779e42c4d03cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4ebed07fef4631babc0a28696ca191" is_pre_election: true
I20260925 01:26:04.907441 20269 raft_consensus.cc:2454] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 227fa41aa4a44b6daea779e42c4d03cc in term 0.
I20260925 01:26:04.907621 20268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "c1dc93724f55467b9d24a03442f3dd72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4ebed07fef4631babc0a28696ca191"
I20260925 01:26:04.907893 20268 raft_consensus.cc:3037] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:04.907935 20338 leader_election.cc:302] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 227fa41aa4a44b6daea779e42c4d03cc, ef4ebed07fef4631babc0a28696ca191; no voters:
I20260925 01:26:04.908330 20485 raft_consensus.cc:2781] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:04.908448 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "c1dc93724f55467b9d24a03442f3dd72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "227fa41aa4a44b6daea779e42c4d03cc"
I20260925 01:26:04.908528 20485 raft_consensus.cc:482] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:04.908638 20485 raft_consensus.cc:3037] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:04.908855 20483 leader_election.cc:288] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [CANDIDATE]: Term 1 election: Requested vote from peers 227fa41aa4a44b6daea779e42c4d03cc (127.6.236.131:43393), ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:04.910928 20268 raft_consensus.cc:2454] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1dc93724f55467b9d24a03442f3dd72 in term 1.
I20260925 01:26:04.911301 20058 leader_election.cc:302] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1dc93724f55467b9d24a03442f3dd72, ef4ebed07fef4631babc0a28696ca191; no voters:
I20260925 01:26:04.911583 20483 raft_consensus.cc:2781] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:04.911504 20485 raft_consensus.cc:504] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.911906 20404 raft_consensus.cc:2379] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c1dc93724f55467b9d24a03442f3dd72 in current term 1: Already voted for candidate 227fa41aa4a44b6daea779e42c4d03cc in this term.
I20260925 01:26:04.912132 20485 leader_election.cc:288] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [CANDIDATE]: Term 1 election: Requested vote from peers c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129:42637), ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179)
I20260925 01:26:04.912518 20125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "227fa41aa4a44b6daea779e42c4d03cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1dc93724f55467b9d24a03442f3dd72"
I20260925 01:26:04.912521 20268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb28515795d0405bbe24a9687f953447" candidate_uuid: "227fa41aa4a44b6daea779e42c4d03cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef4ebed07fef4631babc0a28696ca191"
I20260925 01:26:04.912786 20268 raft_consensus.cc:2379] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 227fa41aa4a44b6daea779e42c4d03cc in current term 1: Already voted for candidate c1dc93724f55467b9d24a03442f3dd72 in this term.
I20260925 01:26:04.913313 20483 raft_consensus.cc:686] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 LEADER]: Becoming Leader. State: Replica: c1dc93724f55467b9d24a03442f3dd72, State: Running, Role: LEADER
I20260925 01:26:04.913794 20483 consensus_queue.cc:237] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } }
I20260925 01:26:04.916225 20336 leader_election.cc:302] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 227fa41aa4a44b6daea779e42c4d03cc; no voters: c1dc93724f55467b9d24a03442f3dd72, ef4ebed07fef4631babc0a28696ca191
I20260925 01:26:04.916620 20485 raft_consensus.cc:2726] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 01:26:04.917757 19970 catalog_manager.cc:5495] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "c1dc93724f55467b9d24a03442f3dd72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dc93724f55467b9d24a03442f3dd72" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 } health_report { overall_health: UNKNOWN } } }
I20260925 01:26:04.988987 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:04.982924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.6.236.129:42637)
0925 01:26:04.983106 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:04.983111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:04.983129 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:26:04.983296 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:04.983300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:04.983376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:04.983498 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:26:04.983818 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.984703 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.986608 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:04.987179 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:04.987273 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:04.987823 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:26:04.988412 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:26:04.988521 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:04.988638 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 01:26:04.988715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:26:04.997164 20268 raft_consensus.cc:1261] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 1 FOLLOWER]: Refusing update from remote peer c1dc93724f55467b9d24a03442f3dd72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:04.997254 20404 raft_consensus.cc:1261] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Refusing update from remote peer c1dc93724f55467b9d24a03442f3dd72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:04.998066 20490 consensus_queue.cc:1035] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "227fa41aa4a44b6daea779e42c4d03cc" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:04.998435 20483 consensus_queue.cc:1035] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef4ebed07fef4631babc0a28696ca191" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 46179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:05.021800 19970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:05.022059 19970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:05.025355 19970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cf76e1b30d0400e91ffe3f4c1fea88e: Sending DeleteTablet for 3 replicas of tablet cb28515795d0405bbe24a9687f953447
I20260925 01:26:05.026353 20105 tablet_service.cc:1514] Processing DeleteTablet for tablet cb28515795d0405bbe24a9687f953447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:05 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:43174
I20260925 01:26:05.026449 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet cb28515795d0405bbe24a9687f953447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:05 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:47394
I20260925 01:26:05.027035 20249 tablet_service.cc:1514] Processing DeleteTablet for tablet cb28515795d0405bbe24a9687f953447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:05 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34570
I20260925 01:26:05.027211 20502 tablet_replica.cc:330] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: stopping tablet replica
I20260925 01:26:05.027495 20503 tablet_replica.cc:330] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: stopping tablet replica
I20260925 01:26:05.027601 20502 raft_consensus.cc:2229] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:26:05.027905 20503 raft_consensus.cc:2229] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:05.028162 20502 raft_consensus.cc:2258] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:05.028313 20503 raft_consensus.cc:2258] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:05.030467 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20009
I20260925 01:26:05.031333 20502 ts_tablet_manager.cc:1905] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:05.034781 20503 ts_tablet_manager.cc:1905] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:05.038312 20502 ts_tablet_manager.cc:1918] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:05.038465 20504 tablet_replica.cc:330] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: stopping tablet replica
I20260925 01:26:05.038507 20502 log.cc:1196] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/cb28515795d0405bbe24a9687f953447
I20260925 01:26:05.038919 20504 raft_consensus.cc:2229] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:05.039105 20502 ts_tablet_manager.cc:1939] T cb28515795d0405bbe24a9687f953447 P c1dc93724f55467b9d24a03442f3dd72: Deleting consensus metadata
I20260925 01:26:05.039423 20504 raft_consensus.cc:2258] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:05.040783 19956 catalog_manager.cc:4841] TS c1dc93724f55467b9d24a03442f3dd72 (127.6.236.129:42637): tablet cb28515795d0405bbe24a9687f953447 (table test-table [id=22202823225e441a9a8eaa9573979044]) successfully deleted
I20260925 01:26:05.043216 20503 ts_tablet_manager.cc:1918] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:05.043372 20503 log.cc:1196] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/cb28515795d0405bbe24a9687f953447
I20260925 01:26:05.043951 20503 ts_tablet_manager.cc:1939] T cb28515795d0405bbe24a9687f953447 P 227fa41aa4a44b6daea779e42c4d03cc: Deleting consensus metadata
I20260925 01:26:05.044981 20504 ts_tablet_manager.cc:1905] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:05.045653 19956 catalog_manager.cc:4841] TS 227fa41aa4a44b6daea779e42c4d03cc (127.6.236.131:43393): tablet cb28515795d0405bbe24a9687f953447 (table test-table [id=22202823225e441a9a8eaa9573979044]) successfully deleted
I20260925 01:26:05.051589 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:05.033861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.6.236.129:42637)
0925 01:26:05.034011 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:05.034017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:05.034039 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:26:05.034274 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:05.034280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:05.034351 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:05.034442 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:05.036284 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.037427 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.039315 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.039951 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.040078 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:05.043519 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:05.043530 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:05.043548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:05.043585 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:05.047655 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.048484 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.048493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.048496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.048654 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.049114 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.049118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.049121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.049579 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:05.049987 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:05.051032 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0925 01:26:05.051258 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:26:05.051626 20504 ts_tablet_manager.cc:1918] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:05.051765 20504 log.cc:1196] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/cb28515795d0405bbe24a9687f953447
I20260925 01:26:05.052152 20504 ts_tablet_manager.cc:1939] T cb28515795d0405bbe24a9687f953447 P ef4ebed07fef4631babc0a28696ca191: Deleting consensus metadata
I20260925 01:26:05.052464 20165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:05.053820 19958 catalog_manager.cc:4841] TS ef4ebed07fef4631babc0a28696ca191 (127.6.236.130:46179): tablet cb28515795d0405bbe24a9687f953447 (table test-table [id=22202823225e441a9a8eaa9573979044]) successfully deleted
I20260925 01:26:05.148131 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20009
I20260925 01:26:05.163548 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20174
I20260925 01:26:05.176640 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:05.164308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.6.236.130:46179)
0925 01:26:05.164496 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:05.164501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:05.164518 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:26:05.164643 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:05.164647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:05.164720 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:05.164798 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:05.166052 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.166813 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.168037 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.168422 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.168517 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:05.171132 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:05.171146 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:05.171163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:05.171195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:05.173929 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.174688 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.174696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.174698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.174833 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.175170 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.175176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.175179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.175646 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:05.175886 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:05.176119 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 01:26:05.176295 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:26:05.177304 20309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:05.267052 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20174
I20260925 01:26:05.281061 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20317
I20260925 01:26:05.294076 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:05.281766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.6.236.131:43393)
0925 01:26:05.281921 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:05.281926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:05.282015 (+ 89us) server_negotiation.cc:407] Connection header received
0925 01:26:05.282160 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:05.282164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:05.282275 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:05.282356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:05.283579 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.284315 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.285491 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:05.285909 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:05.285994 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:05.288552 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:05.288567 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:05.288584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:05.288617 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:05.291181 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.292005 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.292017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.292021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.292178 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:05.292586 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:05.292591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:05.292595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:05.292948 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:05.293234 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:05.293526 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:26:05.293732 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:26:05.294716 20444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:05.393507 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20317
I20260925 01:26:05.409181 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 19939
I20260925 01:26:05.410324 20002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:05.521930 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 19939
I20260925 01:26:05.542740 7090 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:26:05.877030 7090 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:26:05.877228 7090 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:26:06.211972 7090 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:26:06.212 UTC [18990] LOG: received smart shutdown request
2026-09-25 01:26:06.215 UTC [18990] LOG: background worker "logical replication launcher" (PID 18999) exited with exit code 1
2026-09-25 01:26:06.215 UTC [18994] LOG: shutting down
2026-09-25 01:26:06.369 UTC [18990] LOG: database system is shut down
2026-09-25T01:26:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49704 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:26:06 dist-test-slave-s0b2 krb5kdc[20517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:26:06 dist-test-slave-s0b2 krb5kdc[20517](info): set up 2 sockets
Sep 25 01:26:06 dist-test-slave-s0b2 krb5kdc[20517](info): commencing operation
krb5kdc: starting...
W20260925 01:26:07.487154 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:26:07 dist-test-slave-s0b2 krb5kdc[20517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:26:07.713869 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:36667
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:08.026033 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:08.026350 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:08.026490 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:08.033638 20530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:08.033749 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:08.033829 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:08.033903 20530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:08.033965 20530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:08.043708 20530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:08.045004 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:08.046379 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:08.057471 20536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.057453 20538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.057581 20535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:08.058605 20530 server_base.cc:1034] running on GCE node
I20260925 01:26:08.059317 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:08.060672 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:08.061868 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790299568061813 us; error 49 us; skew 500 ppm
I20260925 01:26:08.062204 20530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:08.068472 20530 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:08.069828 20530 webserver.cc:460] Webserver started at http://127.6.236.190:39275/ using document root <none> and password file <none>
I20260925 01:26:08.070523 20530 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:08.070636 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:08.070910 20530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:08.073747 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1640954db9bc4e089a28c31152e1b42d"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "15873da70d7666d705bdde7ee048572f"
server_key_iv: "721136b08b2e0ec33aa93cf857935149"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.074558 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1640954db9bc4e089a28c31152e1b42d"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "15873da70d7666d705bdde7ee048572f"
server_key_iv: "721136b08b2e0ec33aa93cf857935149"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.079319 20530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:26:08.082538 20545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:08.083791 20530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:26:08.083946 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1640954db9bc4e089a28c31152e1b42d"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "15873da70d7666d705bdde7ee048572f"
server_key_iv: "721136b08b2e0ec33aa93cf857935149"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.084131 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:08.098089 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:08.102067 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:08.102389 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:08.119120 20530 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:36667
I20260925 01:26:08.119149 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:36667 every 8 connection(s)
I20260925 01:26:08.120400 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:08.123050 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20530
I20260925 01:26:08.123277 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:26:08.124089 7090 external_mini_cluster.cc:1428] Setting key 3fad178d275c4cfd2f97f454ca627d05
I20260925 01:26:08.124383 20597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:08.132766 20597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.236.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.189@KRBTEST.COM" created.
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:08.157279 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.137271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.6.236.190:36667)
0925 01:26:08.137897 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:08.137904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:08.137960 (+ 56us) server_negotiation.cc:407] Connection header received
0925 01:26:08.139059 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:08.139065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:08.139273 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:08.139449 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:08.140655 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.141862 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.143929 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.144454 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.144576 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.150551 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:08.150559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:08.150572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:08.150615 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:08.153226 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.154017 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.154023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.154025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.154166 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.154618 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.154623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.154625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.155263 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:08.155457 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:08.155980 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 01:26:08.156217 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":278,"threads_started":1}
I20260925 01:26:08.157346 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.135997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:36667 (local address 127.0.0.1:47970)
0925 01:26:08.136731 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.136768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:08.138719 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:08.139492 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:08.139498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:08.140147 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:08.140467 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.140484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.142051 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.142056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.143790 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.143806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.144706 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.144714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.144902 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.145526 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:08.145567 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:08.150314 (+ 4747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:08.153364 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.153376 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.153389 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.153885 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.154328 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.154333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.154336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.154488 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.155390 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:08.155397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:08.155590 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 01:26:08.155793 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 01:26:08.156380 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":92,"thread_start_us":296,"threads_started":1}
I20260925 01:26:08.160326 20597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:26:08.161289 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.160935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53186)
0925 01:26:08.161091 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.161194 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 01:26:08.161526 20548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
I20260925 01:26:08.163100 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.162739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53200)
0925 01:26:08.162925 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.162989 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:26:08.163301 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:45087: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
I20260925 01:26:08.177627 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:45087
--webserver_interface=127.6.236.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:08.212626 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } attempt: 1
I20260925 01:26:08.213739 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.213302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53210)
0925 01:26:08.213486 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.213603 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:26:08.214740 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.214386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53218)
0925 01:26:08.214529 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.214606 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 01:26:08.214908 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:45087: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
I20260925 01:26:08.283231 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } attempt: 2
I20260925 01:26:08.284493 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.283963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53224)
0925 01:26:08.284247 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.284348 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 01:26:08.285637 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.285257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53232)
0925 01:26:08.285424 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.285503 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 01:26:08.285836 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:45087: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
I20260925 01:26:08.377089 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } attempt: 3
I20260925 01:26:08.378304 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.377838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53244)
0925 01:26:08.378036 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.378142 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 01:26:08.379407 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.379022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53258)
0925 01:26:08.379195 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.379267 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 01:26:08.379613 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:45087: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
W20260925 01:26:08.487455 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:08.487733 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:08.487855 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:08.495014 20604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:08.495122 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:08.495210 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:08.495293 20604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:08.495355 20604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:08.504891 20604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.236.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:08.506122 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:08.507400 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:08.518481 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.518530 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.518424 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:08.519483 20604 server_base.cc:1034] running on GCE node
I20260925 01:26:08.520143 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:08.521390 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:08.522576 20604 hybrid_clock.cc:648] HybridClock initialized: now 1790299568522539 us; error 27 us; skew 500 ppm
I20260925 01:26:08.522876 20604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:26:08.522807 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } attempt: 4
I20260925 01:26:08.523824 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.523333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53270)
0925 01:26:08.523609 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.523687 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 01:26:08.524822 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.524470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53278)
0925 01:26:08.524605 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.524690 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:26:08.525023 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:45087: Network error: Client connection negotiation failed: client connection to 127.6.236.189:45087: connect: Connection refused (error 111)
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:08.526031 20604 init.cc:377] Logged in from keytab as kudu/127.6.236.189@KRBTEST.COM (short username kudu)
I20260925 01:26:08.527415 20604 webserver.cc:460] Webserver started at http://127.6.236.189:38893/ using document root <none> and password file <none>
I20260925 01:26:08.528025 20604 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:08.528120 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:08.528352 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:08.531178 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data/instance:
uuid: "0a12ff6575ff47d587bca192b0e30028"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "5165061115f88c8c4ee7b8da256acb63"
server_key_iv: "f04b8fd809472507db348589cc237b17"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.531896 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal/instance:
uuid: "0a12ff6575ff47d587bca192b0e30028"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "5165061115f88c8c4ee7b8da256acb63"
server_key_iv: "f04b8fd809472507db348589cc237b17"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.536562 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:26:08.539758 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:08.540992 20604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:26:08.541141 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "0a12ff6575ff47d587bca192b0e30028"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "5165061115f88c8c4ee7b8da256acb63"
server_key_iv: "f04b8fd809472507db348589cc237b17"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.541293 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:08.558221 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:08.562144 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:08.562397 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:08.578897 20604 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.189:45087
I20260925 01:26:08.578930 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.189:45087 every 8 connection(s)
I20260925 01:26:08.580180 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
I20260925 01:26:08.583792 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:08.587535 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20604
I20260925 01:26:08.587705 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-1/wal/instance
I20260925 01:26:08.588186 7090 external_mini_cluster.cc:1428] Setting key 7b4f2c3b3fd2a6a664cd92f00f40e149
I20260925 01:26:08.590463 20671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.188@KRBTEST.COM" created.
I20260925 01:26:08.611225 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.593127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.6.236.190:36667)
0925 01:26:08.593329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:08.593335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:08.594456 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 01:26:08.596719 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:08.596723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:08.596790 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:08.596882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:08.598761 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.599438 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.601478 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.601881 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.601967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.605777 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:08.605782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:08.605785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:08.605818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:08.608364 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.609047 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.609051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.609053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.609215 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.609604 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.609607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.609609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.609886 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:08.609997 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:08.610697 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 01:26:08.610941 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:26:08.617365 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.593506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:36667 (local address 127.0.0.1:47980)
0925 01:26:08.594286 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.594338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:08.596424 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:08.597142 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:08.597152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:08.597821 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:08.598603 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.598625 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.599605 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.599610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.601360 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.601374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.602097 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.602103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.602376 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.603035 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:08.603061 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:08.605539 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:08.608506 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.608512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.608524 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.608923 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.609309 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.609313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.609315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.609484 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.609983 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:08.609988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:08.610345 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 01:26:08.610564 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 01:26:08.610936 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":315,"threads_started":1}
I20260925 01:26:08.619206 20671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:26:08.633049 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.619777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53292)
0925 01:26:08.620689 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.620712 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:08.620817 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:08.621270 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:08.621275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:08.621713 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:08.621840 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.621848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.623221 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.623226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.624035 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.624046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.624822 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.624828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.625015 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.625542 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:08.625558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:08.627451 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:08.630475 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.630480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.630484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.630896 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.631269 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.631274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.631278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.631445 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.632036 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:08.632041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:08.632173 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 01:26:08.632295 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 01:26:08.632586 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:26:08.633056 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.620063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.6.236.189:45087)
0925 01:26:08.620739 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:08.620747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:08.620777 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:26:08.620929 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:08.620935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:08.621127 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:08.621254 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:08.621987 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.623082 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.624176 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.624685 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.624795 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.627640 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:08.627650 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:08.627664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:08.627695 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:08.630357 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.631015 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.631022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.631024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.631142 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.631548 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.631551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.631553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.631922 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:08.632127 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:08.632449 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 01:26:08.632697 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
I20260925 01:26:08.635346 20671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:26:08.636456 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.635968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36234)
0925 01:26:08.636199 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.636317 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
W20260925 01:26:08.636668 20621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.638012 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:37677
--webserver_interface=127.6.236.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:08.638139 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.637769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36236)
0925 01:26:08.637950 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.638039 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 01:26:08.638403 20671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.687697 20671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 1
I20260925 01:26:08.688863 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.688357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36252)
0925 01:26:08.688576 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.688689 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 01:26:08.690029 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.689550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36260)
0925 01:26:08.689803 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.689889 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260925 01:26:08.690344 20671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.758561 20671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 2
I20260925 01:26:08.759666 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.759198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36270)
0925 01:26:08.759397 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.759485 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:26:08.760761 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.760410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36274)
0925 01:26:08.760555 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.760649 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:26:08.760972 20671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.824246 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } attempt: 5
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:26:08.838068 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.824895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53308)
0925 01:26:08.825116 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.825138 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:08.825278 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:08.825622 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:08.825627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:08.826123 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:08.826398 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.826412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.827384 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.827389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.828775 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:08.828791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.829488 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.829494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:08.829702 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.830388 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:08.830413 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:08.832772 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:08.835755 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.835761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.835765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.836125 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.836482 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:08.836487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:08.836491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:08.836636 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:08.837139 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:08.837144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:08.837301 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 01:26:08.837420 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 01:26:08.837731 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:26:08.838068 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.825141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.6.236.189:45087)
0925 01:26:08.825310 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:08.825315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:08.825337 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:26:08.825400 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:08.825405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:08.825464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:08.825545 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:08.826553 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.827239 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.828902 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:08.829334 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:08.829427 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:08.832959 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:08.832963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:08.832967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:08.832999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:08.835613 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.836244 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.836248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.836251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.836369 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:08.836742 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:08.836746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:08.836748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:08.837021 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:08.837159 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:08.837548 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 01:26:08.837751 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:26:08.839071 20597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:26:08.840018 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.839658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36286)
0925 01:26:08.839818 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.839893 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:26:08.840966 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.840659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36300)
0925 01:26:08.840804 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.840880 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:26:08.841167 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.852171 20671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 3
I20260925 01:26:08.853143 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.852741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36302)
0925 01:26:08.852921 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.853006 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:26:08.854326 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.853838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36316)
0925 01:26:08.854084 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.854159 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260925 01:26:08.854583 20671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.892390 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 1
I20260925 01:26:08.893525 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.893069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36324)
0925 01:26:08.893270 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.893372 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 01:26:08.894685 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.894314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36340)
0925 01:26:08.894467 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.894541 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:26:08.894898 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
W20260925 01:26:08.952198 20678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:08.952485 20678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:08.952623 20678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:08.959710 20678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:08.959821 20678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:08.959903 20678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:08.959981 20678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:08.960044 20678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:08.963096 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 2
I20260925 01:26:08.964169 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.963696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36350)
0925 01:26:08.963924 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.964011 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:26:08.965184 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.964845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36360)
0925 01:26:08.964998 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.965078 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:26:08.965405 20597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:08.969983 20678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:36667,127.6.236.189:45087,127.6.236.188:37677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.236.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:08.971246 20678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:08.972486 20678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:08.983515 20683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.983484 20684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:08.983453 20686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:08.984174 20678 server_base.cc:1034] running on GCE node
I20260925 01:26:08.984925 20678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:08.986297 20678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:08.987506 20678 hybrid_clock.cc:648] HybridClock initialized: now 1790299568987451 us; error 49 us; skew 500 ppm
I20260925 01:26:08.987848 20678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:08 dist-test-slave-s0b2 krb5kdc[20517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:08.990916 20678 init.cc:377] Logged in from keytab as kudu/127.6.236.188@KRBTEST.COM (short username kudu)
I20260925 01:26:08.992344 20678 webserver.cc:460] Webserver started at http://127.6.236.188:36805/ using document root <none> and password file <none>
I20260925 01:26:08.993042 20678 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:08.993155 20678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:08.993441 20678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:08.996313 20678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data/instance:
uuid: "33e7626690834de68dd5227993bb9ab8"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "e406cc708e06b2b93c1d8ece1e44853e"
server_key_iv: "abbe9a2b6f2f1b435086d19b531f888e"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.997114 20678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal/instance:
uuid: "33e7626690834de68dd5227993bb9ab8"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "e406cc708e06b2b93c1d8ece1e44853e"
server_key_iv: "abbe9a2b6f2f1b435086d19b531f888e"
server_key_version: "encryptionkey@0"
I20260925 01:26:08.997794 20671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 4
I20260925 01:26:08.998867 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.998417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36368)
0925 01:26:08.998634 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.998715 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 01:26:08.999855 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:08.999532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36384)
0925 01:26:08.999682 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:08.999753 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:26:09.000041 20671 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:37677: Network error: Client connection negotiation failed: client connection to 127.6.236.188:37677: connect: Connection refused (error 111)
I20260925 01:26:09.002493 20678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 01:26:09.005795 20693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:09.007477 20678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:09.007625 20678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "33e7626690834de68dd5227993bb9ab8"
format_stamp: "Formatted at 2026-09-25 01:26:08 on dist-test-slave-s0b2"
server_key: "e406cc708e06b2b93c1d8ece1e44853e"
server_key_iv: "abbe9a2b6f2f1b435086d19b531f888e"
server_key_version: "encryptionkey@0"
I20260925 01:26:09.007809 20678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:09.018620 20678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:09.022528 20678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:09.022768 20678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:09.038326 20678 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.188:37677
I20260925 01:26:09.038327 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.188:37677 every 8 connection(s)
I20260925 01:26:09.039600 20678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
I20260925 01:26:09.043149 20745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:09.047827 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20678
I20260925 01:26:09.048000 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299375376939-7090-0/minicluster-data/master-2/wal/instance
I20260925 01:26:09.048434 7090 external_mini_cluster.cc:1428] Setting key ce2ce65aa42c98931637a4e4346eaf14
I20260925 01:26:09.049597 20745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:26:09.068898 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.052396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.6.236.190:36667)
0925 01:26:09.052585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.052591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.053116 (+ 525us) server_negotiation.cc:407] Connection header received
0925 01:26:09.053513 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.053519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.053610 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.053711 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.055321 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.056467 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.058436 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.059088 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.059221 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.063118 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.063123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.063126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.063157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.065708 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.066600 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.066604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.066606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.066748 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.067210 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.067215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.067218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.067536 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.067676 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.068410 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.068565 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:26:09.070127 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.052302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:36667 (local address 127.0.0.1:47984)
0925 01:26:09.052961 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:09.053007 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:09.053295 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:09.053814 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:09.053821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:09.054596 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:09.055133 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.055155 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.056621 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.056624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.058273 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.058288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.059298 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.059303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.059532 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.060258 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:09.060294 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:09.062935 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:09.065840 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.065851 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.065862 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.066494 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.066850 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.066854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.066858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.067072 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.067643 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:09.067650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:09.068023 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 01:26:09.068310 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 01:26:09.068982 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":44,"thread_start_us":269,"threads_started":1}
I20260925 01:26:09.070170 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.050161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.6.236.188:37677)
0925 01:26:09.050934 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.050949 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.050995 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:26:09.053035 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.053057 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.053392 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.053589 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.055316 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.056896 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.058696 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.059489 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.059577 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.063824 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.063839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.063850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.063883 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.066840 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.067634 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.067642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.067645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.067790 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.068166 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.068171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.068173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.068576 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.068832 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.069355 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.069585 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":538,"spinlock_wait_cycles":31104,"thread_start_us":297,"threads_started":1}
I20260925 01:26:09.071681 20597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 3
I20260925 01:26:09.071859 20745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:09.086596 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.072450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:45087 (local address 127.0.0.1:53312)
0925 01:26:09.072846 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:09.072861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:09.073007 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:09.073528 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:09.073533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:09.073932 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:09.074108 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.074121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.075477 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.075482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.076613 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.076623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.077485 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.077489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.077617 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.078099 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:09.078121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:09.080553 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:09.083953 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.083958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.083963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.084373 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.084783 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.084786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.084788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.084977 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.085620 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:09.085625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:09.085762 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 01:26:09.085922 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 01:26:09.086182 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":3}
I20260925 01:26:09.086625 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.072551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.6.236.189:45087)
0925 01:26:09.072730 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.072735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.072922 (+ 187us) server_negotiation.cc:407] Connection header received
0925 01:26:09.073201 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.073207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.073283 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.073374 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.074310 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.075297 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.076740 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.077359 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.077474 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.080760 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.080767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.080772 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.080812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.083837 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.084498 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.084504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.084508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.084671 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.085076 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.085081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.085084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.085400 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.085546 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.086037 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.086274 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:26:09.086843 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.072147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36398)
0925 01:26:09.072322 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:09.072339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:09.072433 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:09.072976 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:09.072981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:09.073363 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:09.073587 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.073600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.074778 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.074783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.075815 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.075828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.077672 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.077678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.077843 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.078374 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:09.078395 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:09.080903 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:09.084008 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.084013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.084017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.084586 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.085003 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.085008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.085011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.085155 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.085834 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:09.085837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:09.086021 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 01:26:09.086202 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 01:26:09.086487 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 01:26:09.087597 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.072546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.6.236.188:37677)
0925 01:26:09.072696 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.072701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.072716 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:26:09.072757 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.072762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.072823 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.072911 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.073743 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.074647 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.075967 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.076610 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.076741 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.081090 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.081097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.081102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.081137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.083865 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.084690 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.084696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.084699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.084878 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.085269 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.085275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.085278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.085524 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.085648 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.087096 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.087292 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":54144}
I20260925 01:26:09.088618 20745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:26:09.090858 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.073729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.6.236.190:36667)
0925 01:26:09.073904 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.073909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.073924 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:26:09.073963 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.073968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.074029 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.074131 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.075585 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.076420 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.078291 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.078957 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.079084 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.083746 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.083751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.083756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.083792 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.087408 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.088263 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.088269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.088272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.088431 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.088833 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.088838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.088842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.089137 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.089432 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.090315 (+ 883us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.090519 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:26:09.093217 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: Bootstrap starting.
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:26:09.097218 20597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:09.098443 20597 log.cc:824] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:09.101467 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.089462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36408)
0925 01:26:09.089621 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:09.089637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:09.089728 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:09.090093 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:09.090097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:09.090511 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:09.090630 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.090638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.091533 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.091538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.092365 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.092375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.093169 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.093173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.093313 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.093776 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:09.093797 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:09.095797 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:09.098858 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.098864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.098869 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.099348 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.099689 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.099694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.099698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.099818 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.100434 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:09.100439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:09.100612 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 01:26:09.100779 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:26:09.101077 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:26:09.101641 20597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: No bootstrap required, opened a new log
I20260925 01:26:09.101683 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.089740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.6.236.188:37677)
0925 01:26:09.089883 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.089887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.089898 (+ 11us) server_negotiation.cc:407] Connection header received
0925 01:26:09.089929 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.089932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.089977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.090050 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.090761 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.091407 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.092491 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.092907 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.093014 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.095963 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.095968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.095972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.096004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.098726 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.099463 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.099468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.099471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.099586 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.099917 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.099923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.099926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.100301 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.100457 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.100888 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.101254 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":48}
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:26:09.107434 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: Bootstrap starting.
I20260925 01:26:09.110503 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.095898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.6.236.189:45087)
0925 01:26:09.096080 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.096086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.096103 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:26:09.096148 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.096153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.096220 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.096308 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.097933 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.099004 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.100716 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.101282 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.101398 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.105477 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.105481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.105485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.105516 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.108033 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.108686 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.108697 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.108700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.108832 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.109173 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.109176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.109178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.109430 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.109678 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.109975 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.110186 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:26:09.111791 20745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:09.112169 20597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.112564 20597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:09.112691 20597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1640954db9bc4e089a28c31152e1b42d, State: Initialized, Role: FOLLOWER
I20260925 01:26:09.113214 20745 log.cc:824] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:09.113396 20597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.115646 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:26:09.115670 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.116194 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.116957 20597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:09.117234 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: No bootstrap required, opened a new log
I20260925 01:26:09.128242 20745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.128677 20745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:09.128787 20745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e7626690834de68dd5227993bb9ab8, State: Initialized, Role: FOLLOWER
I20260925 01:26:09.129544 20745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.136440 20760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.136811 20760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.138566 20745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:09.156150 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.137173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.6.236.190:36667)
0925 01:26:09.137349 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.137355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.138167 (+ 812us) server_negotiation.cc:407] Connection header received
0925 01:26:09.138461 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.138467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.138547 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.138621 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.141210 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.142458 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.144837 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.145439 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.145526 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.149044 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.149051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.149056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.149096 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.152580 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.153620 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.153625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.153628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.153787 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.154309 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.154315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.154318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.154697 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.154938 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.155550 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.155789 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:26:09.163228 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.137276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.6.236.188:37677)
0925 01:26:09.137447 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.137453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.145517 (+ 8064us) server_negotiation.cc:407] Connection header received
0925 01:26:09.146253 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.146259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.146341 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.146410 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.148781 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.149801 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.151777 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.152426 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.152522 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.155803 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.155809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.155814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.155852 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.159866 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.160797 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.160802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.160806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.160973 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.161442 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.161448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.161451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.161836 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.162108 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.162653 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.162880 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 01:26:09.170004 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.138333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.6.236.189:45087)
0925 01:26:09.138499 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.138504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.151178 (+ 12674us) server_negotiation.cc:407] Connection header received
0925 01:26:09.151952 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.151960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.152050 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.152117 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.154121 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.155310 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.158742 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.159383 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.159512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.162658 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.162665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.162669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.162715 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.165979 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.166967 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.166972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.166975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.167130 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.167590 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.167595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.167598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.167978 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.168212 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.169350 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.169627 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:26:09.172683 20777 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:26:09.172885 20777 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:26:09.171986 20530 master_runner.cc:186] Error getting master registration for 127.6.236.189:45087: OK, instance_id { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" instance_seqno: 1790299568574040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:26:09.188333 20530 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:26:09.202404 20784 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:26:09.202551 20784 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:26:09.191606 20678 master_runner.cc:186] Error getting master registration for 127.6.236.189:45087: OK, instance_id { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" instance_seqno: 1790299568574040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:26:09.203601 20678 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:26:09.274682 20699 tablet.cc:2369] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:09.299278 20671 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } attempt: 5
Sep 25 01:26:09 dist-test-slave-s0b2 krb5kdc[20517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:26:09.316365 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.300015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:37677 (local address 127.0.0.1:36434)
0925 01:26:09.300303 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:09.300324 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:09.300435 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:09.300850 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:09.300855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:09.301382 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:09.301632 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.301645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.303257 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.303263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.304699 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:09.304716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.305757 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.305763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:09.305998 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.306699 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:09.306724 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:09.309468 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:09.313437 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.313443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.313447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.313981 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.314483 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:09.314489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:09.314492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:09.314668 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:09.315367 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:09.315372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:09.315532 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 01:26:09.315699 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:26:09.315977 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 01:26:09.317519 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:09.300161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.6.236.188:37677)
0925 01:26:09.300342 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:09.300348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:09.300374 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:26:09.300567 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:09.300573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:09.300693 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:09.300805 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:09.301816 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.303057 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.304872 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:09.305590 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:09.305717 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:09.309708 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:09.309716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:09.309721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:09.309759 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:09.313287 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.314147 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.314152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.314156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.314347 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:09.314797 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:09.314802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:09.314806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:09.315163 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:09.315295 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:09.316746 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0925 01:26:09.317110 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:26:09.323333 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: Bootstrap starting.
I20260925 01:26:09.327474 20671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:09.328756 20671 log.cc:824] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:09.332932 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: No bootstrap required, opened a new log
I20260925 01:26:09.343808 20671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.344247 20671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:09.344362 20671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a12ff6575ff47d587bca192b0e30028, State: Initialized, Role: FOLLOWER
I20260925 01:26:09.345033 20671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.346081 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.346566 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.347374 20671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:26:09.355062 20551 tablet.cc:2369] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:09.365358 20604 master_runner.cc:263] No leader master found from master 0a12ff6575ff47d587bca192b0e30028
I20260925 01:26:09.365504 20604 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:26:09.369005 20798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:26:09.369134 20798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:26:09.440092 20760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:09.440441 20760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.442104 20760 leader_election.cc:288] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1640954db9bc4e089a28c31152e1b42d (127.6.236.190:36667), 0a12ff6575ff47d587bca192b0e30028 (127.6.236.189:45087)
I20260925 01:26:09.443408 20572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e7626690834de68dd5227993bb9ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1640954db9bc4e089a28c31152e1b42d" is_pre_election: true
I20260925 01:26:09.444036 20572 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33e7626690834de68dd5227993bb9ab8 in term 0.
I20260925 01:26:09.445118 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e7626690834de68dd5227993bb9ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a12ff6575ff47d587bca192b0e30028" is_pre_election: true
I20260925 01:26:09.445734 20646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33e7626690834de68dd5227993bb9ab8 in term 0.
I20260925 01:26:09.446497 20696 leader_election.cc:302] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1640954db9bc4e089a28c31152e1b42d, 33e7626690834de68dd5227993bb9ab8; no voters:
I20260925 01:26:09.447597 20760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:09.447753 20760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:09.447857 20760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:09.451357 20760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.452517 20572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e7626690834de68dd5227993bb9ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1640954db9bc4e089a28c31152e1b42d"
I20260925 01:26:09.452795 20572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:09.456697 20572 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e7626690834de68dd5227993bb9ab8 in term 1.
I20260925 01:26:09.457237 20696 leader_election.cc:302] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1640954db9bc4e089a28c31152e1b42d, 33e7626690834de68dd5227993bb9ab8; no voters:
I20260925 01:26:09.458935 20760 leader_election.cc:288] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 1640954db9bc4e089a28c31152e1b42d (127.6.236.190:36667), 0a12ff6575ff47d587bca192b0e30028 (127.6.236.189:45087)
I20260925 01:26:09.459496 20760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:09.460117 20760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 33e7626690834de68dd5227993bb9ab8, State: Running, Role: LEADER
I20260925 01:26:09.460894 20760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } }
I20260925 01:26:09.483000 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e7626690834de68dd5227993bb9ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a12ff6575ff47d587bca192b0e30028"
I20260925 01:26:09.483292 20646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:09.486866 20646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e7626690834de68dd5227993bb9ab8 in term 1.
I20260925 01:26:09.488623 20800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e7626690834de68dd5227993bb9ab8. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.489162 20800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:09.496239 20802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:09.498879 20802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:09.520262 20646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [term 1 FOLLOWER]: Refusing update from remote peer 33e7626690834de68dd5227993bb9ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:09.521477 20800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:09.528903 20572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [term 1 FOLLOWER]: Refusing update from remote peer 33e7626690834de68dd5227993bb9ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:09.529886 20800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:26:09.564726 20625 tablet.cc:2369] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:09.566135 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e7626690834de68dd5227993bb9ab8. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.566463 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.568773 20803 mvcc.cc:205] Tried to move back new op lower bound from 7333067036739563520 to 7333067036543942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333067036739563520}]
I20260925 01:26:09.572436 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e7626690834de68dd5227993bb9ab8. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.572779 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.574431 20805 mvcc.cc:205] Tried to move back new op lower bound from 7333067036739563520 to 7333067036543942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333067036739563520}]
I20260925 01:26:09.579667 20760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.579984 20760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:09.580264 20800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.580502 20800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:09.584728 20802 catalog_manager.cc:1347] Generated new cluster ID: b69509164de540a4a18df6ce13520722
I20260925 01:26:09.584820 20802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:09.587478 20804 mvcc.cc:205] Tried to move back new op lower bound from 7333067036739563520 to 7333067036543942656. Current Snapshot: MvccSnapshot[applied={T|T < 7333067036739563520}]
I20260925 01:26:09.590811 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.591079 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.600314 20802 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:26:09.603667 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e7626690834de68dd5227993bb9ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1640954db9bc4e089a28c31152e1b42d" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 36667 } } peers { permanent_uuid: "0a12ff6575ff47d587bca192b0e30028" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 45087 } } peers { permanent_uuid: "33e7626690834de68dd5227993bb9ab8" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 37677 } } }
I20260925 01:26:09.603911 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:26:09.606467 20802 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:09.622112 20802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e7626690834de68dd5227993bb9ab8: Generated new TSK 0
I20260925 01:26:09.623214 20802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:26:10.174904 20777 catalog_manager.cc:1259] Loaded cluster ID: b69509164de540a4a18df6ce13520722
I20260925 01:26:10.175009 20777 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: loading cluster ID for follower catalog manager: success
I20260925 01:26:10.177379 20777 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: acquiring CA information for follower catalog manager: success
I20260925 01:26:10.179159 20777 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1640954db9bc4e089a28c31152e1b42d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:26:10.371312 20798 catalog_manager.cc:1259] Loaded cluster ID: b69509164de540a4a18df6ce13520722
I20260925 01:26:10.371410 20798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: loading cluster ID for follower catalog manager: success
I20260925 01:26:10.373795 20798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: acquiring CA information for follower catalog manager: success
I20260925 01:26:10.375525 20798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a12ff6575ff47d587bca192b0e30028: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:26:10.712783 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20530
I20260925 01:26:10.713948 20592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:10.822878 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20530
I20260925 01:26:10.840992 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20604
I20260925 01:26:10.842199 20666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:10.946669 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20604
I20260925 01:26:10.963564 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20678
I20260925 01:26:10.964582 20740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:11.069733 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20678
2026-09-25T01:26:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4594 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:26:11 dist-test-slave-s0b2 krb5kdc[20820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:26:11 dist-test-slave-s0b2 krb5kdc[20820](info): set up 2 sockets
Sep 25 01:26:11 dist-test-slave-s0b2 krb5kdc[20820](info): commencing operation
krb5kdc: starting...
W20260925 01:26:12.079470 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:26:12.294268 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35861
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:35861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:12.608045 20833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:12.608390 20833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:12.608536 20833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:12.615686 20833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:12.615801 20833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:12.615880 20833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:12.615957 20833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:12.616019 20833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:12.625883 20833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:35861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:12.627199 20833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:12.628474 20833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:12.639619 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:12.639578 20841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:12.639740 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:12.640378 20833 server_base.cc:1034] running on GCE node
I20260925 01:26:12.641089 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:12.642498 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:12.643719 20833 hybrid_clock.cc:648] HybridClock initialized: now 1790299572643680 us; error 28 us; skew 500 ppm
I20260925 01:26:12.644054 20833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:12.647081 20833 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:12.648497 20833 webserver.cc:460] Webserver started at http://127.6.236.190:40627/ using document root <none> and password file <none>
I20260925 01:26:12.649169 20833 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:12.649278 20833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:12.649577 20833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:12.652531 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "abec2b92c06841dab96ec0e9bf3716ca"
format_stamp: "Formatted at 2026-09-25 01:26:12 on dist-test-slave-s0b2"
server_key: "2e0377cf4683829b16b00dbcc4e90e57"
server_key_iv: "d7bbd264b82e3b9f1a6ee1320ae18ec9"
server_key_version: "encryptionkey@0"
I20260925 01:26:12.653335 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "abec2b92c06841dab96ec0e9bf3716ca"
format_stamp: "Formatted at 2026-09-25 01:26:12 on dist-test-slave-s0b2"
server_key: "2e0377cf4683829b16b00dbcc4e90e57"
server_key_iv: "d7bbd264b82e3b9f1a6ee1320ae18ec9"
server_key_version: "encryptionkey@0"
I20260925 01:26:12.658344 20833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:26:12.661654 20848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:12.662979 20833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:26:12.663128 20833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "abec2b92c06841dab96ec0e9bf3716ca"
format_stamp: "Formatted at 2026-09-25 01:26:12 on dist-test-slave-s0b2"
server_key: "2e0377cf4683829b16b00dbcc4e90e57"
server_key_iv: "d7bbd264b82e3b9f1a6ee1320ae18ec9"
server_key_version: "encryptionkey@0"
I20260925 01:26:12.663313 20833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:12.682904 20833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:12.687050 20833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:12.687323 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:12.703780 20833 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:35861
I20260925 01:26:12.703805 20899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:35861 every 8 connection(s)
I20260925 01:26:12.705109 20833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:12.708716 20900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:12.714553 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20833
I20260925 01:26:12.714780 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:26:12.715374 7090 external_mini_cluster.cc:1428] Setting key 04295de56ca9a8b13c9a2796eec3247d
I20260925 01:26:12.718961 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca: Bootstrap starting.
I20260925 01:26:12.722563 20900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:12.723539 20900 log.cc:824] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:12.726584 20900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca: No bootstrap required, opened a new log
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:12.735821 20900 raft_consensus.cc:348] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } }
I20260925 01:26:12.736274 20900 raft_consensus.cc:374] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:12.736387 20900 raft_consensus.cc:729] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abec2b92c06841dab96ec0e9bf3716ca, State: Initialized, Role: FOLLOWER
I20260925 01:26:12.736987 20900 consensus_queue.cc:260] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } }
I20260925 01:26:12.737310 20900 raft_consensus.cc:388] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:12.737460 20900 raft_consensus.cc:482] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:12.737632 20900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:12.738492 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:12.717123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.6.236.190:35861)
0925 01:26:12.717863 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:12.717874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:12.717920 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:26:12.719648 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:12.719672 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:12.720041 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:12.720413 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:12.722199 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:12.723589 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:12.725265 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:12.726296 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:12.726471 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:12.730405 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:12.730416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:12.730434 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:12.730498 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:12.734298 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:12.734962 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:12.734971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:12.734980 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:12.735181 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:12.735514 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:12.735519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:12.735522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:12.736218 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:12.736508 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:12.736967 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 01:26:12.737381 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":343,"threads_started":1}
I20260925 01:26:12.740008 20900 raft_consensus.cc:504] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } }
I20260925 01:26:12.740469 20900 leader_election.cc:302] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abec2b92c06841dab96ec0e9bf3716ca; no voters:
I20260925 01:26:12.741065 20900 leader_election.cc:288] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:26:12.741146 20905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:12.742399 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:26:12.742352 20905 raft_consensus.cc:686] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [term 1 LEADER]: Becoming Leader. State: Replica: abec2b92c06841dab96ec0e9bf3716ca, State: Running, Role: LEADER
I20260925 01:26:12.742818 20905 consensus_queue.cc:237] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } }
I20260925 01:26:12.743158 20900 sys_catalog.cc:564] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:12.745407 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader abec2b92c06841dab96ec0e9bf3716ca. Latest consensus state: current_term: 1 leader_uuid: "abec2b92c06841dab96ec0e9bf3716ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } } }
I20260925 01:26:12.745873 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:12.745398 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abec2b92c06841dab96ec0e9bf3716ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abec2b92c06841dab96ec0e9bf3716ca" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35861 } } }
I20260925 01:26:12.746098 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:12.747171 20911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:12.752717 20911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:12.761189 20911 catalog_manager.cc:1347] Generated new cluster ID: c29aaba80c0742068ad1e730f9425e89
I20260925 01:26:12.761286 20911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:12.774451 20911 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:26:12.775683 20911 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:12.795591 20911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abec2b92c06841dab96ec0e9bf3716ca: Generated new TSK 0
I20260925 01:26:12.796389 20911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:26:12.803352 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20833
I20260925 01:26:12.804436 20895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:12.894634 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20833
2026-09-25T01:26:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1820 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20928](info): set up 2 sockets
Sep 25 01:26:12 dist-test-slave-s0b2 krb5kdc[20928](info): commencing operation
krb5kdc: starting...
W20260925 01:26:13.919337 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:26:14 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:26:14.135977 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:14.446025 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:14.446354 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:14.446501 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:14.454676 20941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:14.454789 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:14.454872 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:14.454948 20941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:14.455008 20941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:14.465008 20941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:14.466343 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:14.467619 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:14.478629 20946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:14.478749 20947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:14.478698 20949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:14.479704 20941 server_base.cc:1034] running on GCE node
I20260925 01:26:14.480470 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:14.481827 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:14.483024 20941 hybrid_clock.cc:648] HybridClock initialized: now 1790299574482976 us; error 33 us; skew 500 ppm
I20260925 01:26:14.483363 20941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:14 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:14.490648 20941 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:14.492144 20941 webserver.cc:460] Webserver started at http://127.6.236.190:36915/ using document root <none> and password file <none>
I20260925 01:26:14.492820 20941 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:14.492934 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:14.493225 20941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:14.496232 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "874e75dc42e7459aa823d025ea1d7ce8"
format_stamp: "Formatted at 2026-09-25 01:26:14 on dist-test-slave-s0b2"
server_key: "72a4d7ac722d5abf66f4e1285d27726b"
server_key_iv: "791c20ca57d4fe7e63ddf4e7e8dafc19"
server_key_version: "encryptionkey@0"
I20260925 01:26:14.496951 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "874e75dc42e7459aa823d025ea1d7ce8"
format_stamp: "Formatted at 2026-09-25 01:26:14 on dist-test-slave-s0b2"
server_key: "72a4d7ac722d5abf66f4e1285d27726b"
server_key_iv: "791c20ca57d4fe7e63ddf4e7e8dafc19"
server_key_version: "encryptionkey@0"
I20260925 01:26:14.501991 20941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:26:14.505390 20956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:14.506778 20941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:14.506951 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "874e75dc42e7459aa823d025ea1d7ce8"
format_stamp: "Formatted at 2026-09-25 01:26:14 on dist-test-slave-s0b2"
server_key: "72a4d7ac722d5abf66f4e1285d27726b"
server_key_iv: "791c20ca57d4fe7e63ddf4e7e8dafc19"
server_key_version: "encryptionkey@0"
I20260925 01:26:14.507133 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:14.518676 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:14.522526 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:14.522790 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:14.539899 20941 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46041
I20260925 01:26:14.539918 21007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46041 every 8 connection(s)
I20260925 01:26:14.541308 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:14.545188 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:14.545315 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 20941
I20260925 01:26:14.545532 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:26:14.546170 7090 external_mini_cluster.cc:1428] Setting key 588efd86580770954cdecb02770d5841
I20260925 01:26:14.559158 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap starting.
Sep 25 01:26:14 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:14.562435 21008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:14.563532 21008 log.cc:824] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:14.566402 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: No bootstrap required, opened a new log
I20260925 01:26:14.570464 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:14.548260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.6.236.190:46041)
0925 01:26:14.549050 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:14.549065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:14.549119 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:26:14.551632 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:14.551661 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:14.552177 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:14.552644 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:14.553934 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:14.555956 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:14.557363 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:14.558397 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:14.558569 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:14.562762 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:14.562771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:14.562789 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:14.562834 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:14.565839 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:14.566611 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:14.566619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:14.566628 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:14.566842 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:14.567246 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:14.567250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:14.567253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:14.567951 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:14.568274 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:14.568607 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:26:14.569072 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":320,"threads_started":1}
I20260925 01:26:14.575388 21008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:14.575791 21008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:14.575867 21008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Initialized, Role: FOLLOWER
I20260925 01:26:14.576334 21008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:14.576520 21008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:14.576611 21008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:14.576735 21008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:14.578804 21008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:14.579227 21008 leader_election.cc:302] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874e75dc42e7459aa823d025ea1d7ce8; no voters:
I20260925 01:26:14.579769 21008 leader_election.cc:288] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:26:14.579890 21013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:14.580930 21013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Running, Role: LEADER
I20260925 01:26:14.581406 21013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:14.581872 21008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:14.584201 21015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874e75dc42e7459aa823d025ea1d7ce8. Latest consensus state: current_term: 1 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:14.584173 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:14.584568 21015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:14.584585 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:14.585840 21018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:14.590658 21018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:14.599412 21018 catalog_manager.cc:1347] Generated new cluster ID: 6f403c2cc004401182e929b4793fc0d8
I20260925 01:26:14.599516 21018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:14.615954 21018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:26:14.617813 21018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:14.626971 21018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Generated new TSK 0
I20260925 01:26:14.627981 21018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:26:14.686383 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:15.006729 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:15.007019 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:15.007153 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:15.014499 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:15.014629 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:15.014788 21034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:26:15.024319 21034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:15.025591 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:15.027034 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:15.039628 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.039723 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.039819 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:15.040671 21034 server_base.cc:1034] running on GCE node
I20260925 01:26:15.041241 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:15.042114 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:15.043310 21034 hybrid_clock.cc:648] HybridClock initialized: now 1790299575043246 us; error 53 us; skew 500 ppm
I20260925 01:26:15.043586 21034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:15 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:15.046465 21034 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:26:15.047775 21034 webserver.cc:460] Webserver started at http://127.6.236.129:35023/ using document root <none> and password file <none>
I20260925 01:26:15.048395 21034 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:15.048506 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:15.048790 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:15.051564 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "6914d11c690c4be79eb382496e22e2ad"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "26b9e597080d02e746e02f59c9dbcd7b"
server_key_iv: "e418c9e235d277beb10ac1e1f5997662"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.052275 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "6914d11c690c4be79eb382496e22e2ad"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "26b9e597080d02e746e02f59c9dbcd7b"
server_key_iv: "e418c9e235d277beb10ac1e1f5997662"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.057088 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:26:15.060501 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.061764 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:26:15.061908 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "6914d11c690c4be79eb382496e22e2ad"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "26b9e597080d02e746e02f59c9dbcd7b"
server_key_iv: "e418c9e235d277beb10ac1e1f5997662"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.062080 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:15.080906 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:15.084810 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:15.085062 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:15.085944 21034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:15.087318 21034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:15.087395 21034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.087522 21034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:15.087575 21034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.118129 21034 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:33505
I20260925 01:26:15.118191 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:33505 every 8 connection(s)
I20260925 01:26:15.119379 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:26:15.120326 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21034
I20260925 01:26:15.120546 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:26:15.121142 7090 external_mini_cluster.cc:1428] Setting key 0c93cfbd222728cd6cca0573e3f1e751
Sep 25 01:26:15 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:15.139993 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:15.122018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42651 (local address 127.6.236.190:46041)
0925 01:26:15.122523 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:15.122529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:15.123550 (+ 1021us) server_negotiation.cc:407] Connection header received
0925 01:26:15.125297 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:15.125302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:15.125380 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:15.125473 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:15.127047 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.128028 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.130072 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.130577 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.130672 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:15.134211 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:15.134217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:15.134222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:15.134286 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:15.137130 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:15.137794 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:15.137798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:15.137801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:15.137936 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:15.138312 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:15.138316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:15.138320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:15.138653 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:15.138788 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:15.139434 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 01:26:15.139667 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260925 01:26:15.140684 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:15.122605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:42651)
0925 01:26:15.123393 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:15.123430 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:15.125028 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:15.125735 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:15.125743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:15.126316 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:15.126873 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:15.126891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.128199 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.128204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:15.129920 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:15.129935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.130805 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.130813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:15.131086 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:15.131786 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:15.131812 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:15.133990 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:15.137265 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:15.137272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:15.137283 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:15.137679 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:15.138025 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:15.138028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:15.138030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:15.138200 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:15.138760 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:15.138765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:15.139068 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 01:26:15.139281 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 01:26:15.139643 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":278,"threads_started":1}
I20260925 01:26:15.142319 21162 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:15.142598 21162 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:15.143297 21162 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:26:15.145305 20973 ts_manager.cc:194] Registered new tserver with Master: 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
I20260925 01:26:15.146909 20973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:42651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:26:15.173396 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:15.483955 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:15.484215 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:15.484344 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:15.491396 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:15.491492 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:15.491634 21168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:15.500834 21168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:15.502063 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:15.503355 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:15.515811 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.515888 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.515858 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:15.516773 21168 server_base.cc:1034] running on GCE node
I20260925 01:26:15.517345 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:15.518230 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:15.519491 21168 hybrid_clock.cc:648] HybridClock initialized: now 1790299575519435 us; error 43 us; skew 500 ppm
I20260925 01:26:15.519809 21168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:15 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:15.522853 21168 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:15.524204 21168 webserver.cc:460] Webserver started at http://127.6.236.130:44949/ using document root <none> and password file <none>
I20260925 01:26:15.524865 21168 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:15.524977 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:15.525265 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:15.528227 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "2c2979aba9db4b33aed65f8666a6e38f"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "c81912d71b592c28994542b6fa3a6d7f"
server_key_iv: "b960f38ce3e728cd0fe714e1edbdf9d3"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.528939 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "2c2979aba9db4b33aed65f8666a6e38f"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "c81912d71b592c28994542b6fa3a6d7f"
server_key_iv: "b960f38ce3e728cd0fe714e1edbdf9d3"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.533851 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:26:15.537103 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.538511 21168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:26:15.538662 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "2c2979aba9db4b33aed65f8666a6e38f"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "c81912d71b592c28994542b6fa3a6d7f"
server_key_iv: "b960f38ce3e728cd0fe714e1edbdf9d3"
server_key_version: "encryptionkey@0"
I20260925 01:26:15.538841 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:15.554880 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:15.558944 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:15.559203 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:15.560143 21168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:15.561434 21168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:15.561508 21168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.561650 21168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:15.561704 21168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:15.591285 21168 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36703
I20260925 01:26:15.591306 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36703 every 8 connection(s)
I20260925 01:26:15.592489 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:15.593554 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21168
I20260925 01:26:15.593772 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:26:15.594300 7090 external_mini_cluster.cc:1428] Setting key e23338fd31730602b36f689cd0104755
Sep 25 01:26:15 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:15.612674 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:15.595222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:53037 (local address 127.6.236.190:46041)
0925 01:26:15.595423 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:15.595428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:15.596681 (+ 1253us) server_negotiation.cc:407] Connection header received
0925 01:26:15.598470 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:15.598476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:15.598567 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:15.598667 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:15.600099 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.601047 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.602682 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.603233 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.603345 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:15.607172 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:15.607177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:15.607181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:15.607213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:15.609633 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:15.610460 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:15.610464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:15.610466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:15.610622 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:15.611065 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:15.611068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:15.611070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:15.611361 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:15.611480 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:15.612248 (+ 768us) server_negotiation.cc:299] Negotiation successful
0925 01:26:15.612419 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:26:15.614037 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:15.595658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:53037)
0925 01:26:15.596498 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:15.596546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:15.598174 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:15.598863 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:15.598871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:15.599359 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:15.599917 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:15.599935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.601208 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.601211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:15.602548 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:15.602559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:15.603464 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:15.603472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:15.603689 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:15.604330 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:15.604362 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:15.606924 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:15.609766 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:15.609775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:15.609791 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:15.610348 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:15.610724 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:15.610728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:15.610732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:15.610971 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:15.611476 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:15.611484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:15.611885 (+ 401us) client_negotiation.cc:769] Sending connection context
0925 01:26:15.612177 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 01:26:15.612688 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":341,"threads_started":1}
I20260925 01:26:15.615518 21296 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:15.615844 21296 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:15.616726 21296 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
I20260925 01:26:15.618297 20973 ts_manager.cc:194] Registered new tserver with Master: 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:15.619068 20973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:53037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:26:15.646168 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:15.955284 21301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:15.955542 21301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:15.955668 21301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:15.962875 21301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:15.962980 21301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:15.963121 21301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:26:15.972822 21301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:15.974146 21301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:15.975584 21301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:15.988451 21309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.988459 21307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:15.988668 21306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:15.989177 21301 server_base.cc:1034] running on GCE node
I20260925 01:26:15.989804 21301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:15.990803 21301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:15.992004 21301 hybrid_clock.cc:648] HybridClock initialized: now 1790299575991962 us; error 32 us; skew 500 ppm
I20260925 01:26:15.992303 21301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:15 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:15.995340 21301 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:15.996755 21301 webserver.cc:460] Webserver started at http://127.6.236.131:38743/ using document root <none> and password file <none>
I20260925 01:26:15.997437 21301 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:15.997550 21301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:15.997859 21301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:16.000739 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "53076187bcd74d649c3b79fecaaf6892"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "1fa3a9fdbedc1ae3e3a6bb97fcafcb1f"
server_key_iv: "384e511e9534ca2a15be5e4dda083336"
server_key_version: "encryptionkey@0"
I20260925 01:26:16.001462 21301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "53076187bcd74d649c3b79fecaaf6892"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "1fa3a9fdbedc1ae3e3a6bb97fcafcb1f"
server_key_iv: "384e511e9534ca2a15be5e4dda083336"
server_key_version: "encryptionkey@0"
I20260925 01:26:16.006618 21301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:26:16.009999 21316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:16.011477 21301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:16.011620 21301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "53076187bcd74d649c3b79fecaaf6892"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "1fa3a9fdbedc1ae3e3a6bb97fcafcb1f"
server_key_iv: "384e511e9534ca2a15be5e4dda083336"
server_key_version: "encryptionkey@0"
I20260925 01:26:16.011817 21301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:16.031394 21301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:16.035267 21301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:16.035485 21301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:16.036434 21301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:16.037719 21301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:16.037792 21301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:16.037910 21301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:16.037963 21301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:16.067790 21301 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:46263
I20260925 01:26:16.067824 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:46263 every 8 connection(s)
I20260925 01:26:16.069003 21301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:16.075861 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21301
I20260925 01:26:16.076031 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:26:16.076493 7090 external_mini_cluster.cc:1428] Setting key 358983d794f630c9c98c91bdd685e135
Sep 25 01:26:16 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:16.088248 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.071588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:57537 (local address 127.6.236.190:46041)
0925 01:26:16.071826 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.071832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.072907 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 01:26:16.074671 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.074676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.074745 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.074829 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:16.076402 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.077334 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.079066 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.079579 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.079676 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.083102 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:16.083108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:16.083111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:16.083144 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:16.085549 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.086224 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.086227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.086229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.086374 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.086797 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.086802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.086805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.087069 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:16.087177 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.087789 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.087977 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 01:26:16.089179 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.072030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:57537)
0925 01:26:16.072742 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.072786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.074427 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.075086 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.075095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.075601 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:16.076205 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.076230 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.077480 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.077485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.078929 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.078941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.079791 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.079796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.079999 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.080633 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:16.080659 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:16.082901 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:16.085690 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:16.085696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:16.085709 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:16.086120 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:16.086472 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:16.086477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:16.086480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:16.086684 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:16.087175 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:16.087180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:16.087473 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.087722 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.088142 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":262,"threads_started":1}
I20260925 01:26:16.090649 21429 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:16.090969 21429 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:16.091801 21429 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
I20260925 01:26:16.093097 20973 ts_manager.cc:194] Registered new tserver with Master: 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263)
I20260925 01:26:16.093855 20973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:57537
I20260925 01:26:16.103726 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:26:16.120370 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.107983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.6.236.190:46041)
0925 01:26:16.108158 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.108162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.108567 (+ 405us) server_negotiation.cc:407] Connection header received
0925 01:26:16.108698 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.108702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.108759 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.108837 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:16.110187 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.111100 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.112483 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.112894 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.112996 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.115257 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:16.115262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:16.115265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:16.115296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:16.117855 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.118618 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.118622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.118624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.118750 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.119155 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.119159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.119161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.119458 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:16.119667 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.119980 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.120136 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:26:16.124423 20973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:26:16.131096 20973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:26:16.149712 21162 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:16.165843 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.142919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.6.236.130:36703)
0925 01:26:16.157347 (+ 14428us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.157357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.157396 (+ 39us) server_negotiation.cc:407] Connection header received
0925 01:26:16.157518 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.157527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.157856 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.158102 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:16.158975 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.160473 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.163185 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.164636 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.164774 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.164969 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.165353 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.165513 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14215,"thread_start_us":15391,"threads_started":1}
I20260925 01:26:16.165872 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.142458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36703 (local address 127.0.0.1:57170)
0925 01:26:16.151178 (+ 8720us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.151207 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.151315 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.158181 (+ 6866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.158191 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.158229 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:16.158736 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.158753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.160633 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.160636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.163013 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.163030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.164874 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.164882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.165073 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.165123 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.165257 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.165445 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8602,"thread_start_us":16264,"threads_started":1}
I20260925 01:26:16.166823 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.142398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:46263 (local address 127.0.0.1:35616)
0925 01:26:16.143139 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.143177 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.143374 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.160280 (+ 16906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.160285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.160310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:16.160776 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.160791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.162757 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.162762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.164281 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.164293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.166063 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.166069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.166253 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.166287 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.166348 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.166493 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":669,"threads_started":1}
I20260925 01:26:16.167584 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.142646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.6.236.131:46263)
0925 01:26:16.159757 (+ 17111us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.159764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.159798 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:26:16.159888 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.159896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.160084 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.160250 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:16.161017 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.162601 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.164456 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.165845 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.166856 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.167074 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.167136 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.167284 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16953,"thread_start_us":16818,"threads_started":1}
I20260925 01:26:16.168251 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.143961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:33505 (local address 127.0.0.1:43450)
0925 01:26:16.159049 (+ 15088us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.159073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.159216 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.160536 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.160541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.160566 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:16.161088 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.161097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.162764 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.162769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.164929 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.164944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.167558 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.167565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.167720 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.167749 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.167840 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.167954 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14926,"thread_start_us":6966,"threads_started":1}
I20260925 01:26:16.168382 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.144289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.6.236.129:33505)
0925 01:26:16.157856 (+ 13567us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.157862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.159849 (+ 1987us) server_negotiation.cc:407] Connection header received
0925 01:26:16.159958 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.159966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.160180 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.160361 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:16.161252 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.162578 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.165065 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.167140 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.167275 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.167433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.167903 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.168060 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13419,"thread_start_us":13257,"threads_started":1}
I20260925 01:26:16.170380 21231 tablet_service.cc:1467] Processing CreateTablet for tablet d8f67415f49148e28b751801674bf1ed (DEFAULT_TABLE table=test-table [id=8e94926c28df4812bc90538887ab6e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:16.171007 21364 tablet_service.cc:1467] Processing CreateTablet for tablet d8f67415f49148e28b751801674bf1ed (DEFAULT_TABLE table=test-table [id=8e94926c28df4812bc90538887ab6e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:16.171877 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f67415f49148e28b751801674bf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:16.172133 21364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f67415f49148e28b751801674bf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:16.171856 21097 tablet_service.cc:1467] Processing CreateTablet for tablet d8f67415f49148e28b751801674bf1ed (DEFAULT_TABLE table=test-table [id=8e94926c28df4812bc90538887ab6e02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:16.173123 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f67415f49148e28b751801674bf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:16.183368 21448 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Bootstrap starting.
I20260925 01:26:16.186604 21448 tablet_bootstrap.cc:654] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:16.187009 21449 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Bootstrap starting.
I20260925 01:26:16.187978 21448 log.cc:824] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:16.189308 21450 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Bootstrap starting.
I20260925 01:26:16.191075 21449 tablet_bootstrap.cc:654] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:16.191479 21448 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: No bootstrap required, opened a new log
I20260925 01:26:16.191713 21448 ts_tablet_manager.cc:1397] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260925 01:26:16.192256 21449 log.cc:824] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:16.192593 21450 tablet_bootstrap.cc:654] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:16.193821 21450 log.cc:824] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:16.196205 21449 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: No bootstrap required, opened a new log
I20260925 01:26:16.196465 21449 ts_tablet_manager.cc:1397] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260925 01:26:16.196708 21450 tablet_bootstrap.cc:492] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: No bootstrap required, opened a new log
I20260925 01:26:16.196941 21450 ts_tablet_manager.cc:1397] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 01:26:16.200699 21448 raft_consensus.cc:348] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.201133 21448 raft_consensus.cc:374] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:16.201279 21448 raft_consensus.cc:729] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53076187bcd74d649c3b79fecaaf6892, State: Initialized, Role: FOLLOWER
I20260925 01:26:16.201924 21448 consensus_queue.cc:260] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.203022 21429 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:16.203641 21448 ts_tablet_manager.cc:1428] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 01:26:16.204537 21449 raft_consensus.cc:348] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.205078 21449 raft_consensus.cc:374] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:16.205233 21449 raft_consensus.cc:729] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2979aba9db4b33aed65f8666a6e38f, State: Initialized, Role: FOLLOWER
I20260925 01:26:16.205960 21449 consensus_queue.cc:260] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.206498 21450 raft_consensus.cc:348] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.207088 21450 raft_consensus.cc:374] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:16.207093 21296 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:16.207242 21450 raft_consensus.cc:729] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6914d11c690c4be79eb382496e22e2ad, State: Initialized, Role: FOLLOWER
I20260925 01:26:16.207718 21449 ts_tablet_manager.cc:1428] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260925 01:26:16.207861 21450 consensus_queue.cc:260] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.209216 21450 ts_tablet_manager.cc:1428] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 01:26:16.290766 21456 raft_consensus.cc:482] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:16.291054 21456 raft_consensus.cc:504] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.292729 21456 leader_election.cc:288] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263), 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:16.298931 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.293019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55993 (local address 127.6.236.131:46263)
0925 01:26:16.293209 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.293216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.293879 (+ 663us) server_negotiation.cc:407] Connection header received
0925 01:26:16.294064 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.294068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.294129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.294215 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:16.294777 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.295510 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.297256 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.297991 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.298138 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.298222 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.298573 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.298696 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:26:16.299044 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.293331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42055 (local address 127.6.236.130:36703)
0925 01:26:16.293530 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.293536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.294104 (+ 568us) server_negotiation.cc:407] Connection header received
0925 01:26:16.294295 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.294301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.294364 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.294438 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:16.294988 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.295685 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.297381 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.298129 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.298300 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.298370 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.298676 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.298795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:26:16.299067 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.292970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:46263 (local address 127.6.236.129:55993)
0925 01:26:16.293754 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.293774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.293920 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.294266 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.294270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.294302 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:16.294632 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.294640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.295643 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.295648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.297090 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.297103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.298193 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.298198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.298371 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.298439 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.298519 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.298637 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":30,"thread_start_us":437,"threads_started":1}
I20260925 01:26:16.299217 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.293173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36703 (local address 127.6.236.129:42055)
0925 01:26:16.293992 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:16.294005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:16.294135 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:16.294477 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:16.294481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:16.294509 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:16.294820 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.294830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.295829 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.295833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.297223 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:16.297235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.298366 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.298372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:16.298525 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.298547 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:26:16.298626 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 01:26:16.298742 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":6,"thread_start_us":501,"threads_started":1}
I20260925 01:26:16.300104 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f67415f49148e28b751801674bf1ed" candidate_uuid: "6914d11c690c4be79eb382496e22e2ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2979aba9db4b33aed65f8666a6e38f" is_pre_election: true
I20260925 01:26:16.300112 21384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f67415f49148e28b751801674bf1ed" candidate_uuid: "6914d11c690c4be79eb382496e22e2ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53076187bcd74d649c3b79fecaaf6892" is_pre_election: true
I20260925 01:26:16.300541 21251 raft_consensus.cc:2454] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6914d11c690c4be79eb382496e22e2ad in term 0.
I20260925 01:26:16.300541 21384 raft_consensus.cc:2454] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6914d11c690c4be79eb382496e22e2ad in term 0.
I20260925 01:26:16.301151 21051 leader_election.cc:302] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53076187bcd74d649c3b79fecaaf6892, 6914d11c690c4be79eb382496e22e2ad; no voters:
I20260925 01:26:16.301527 21456 raft_consensus.cc:2781] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:16.301698 21456 raft_consensus.cc:482] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:16.301796 21456 raft_consensus.cc:3037] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:16.303875 21456 raft_consensus.cc:504] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.304438 21456 leader_election.cc:288] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [CANDIDATE]: Term 1 election: Requested vote from peers 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263), 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:16.304844 21384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f67415f49148e28b751801674bf1ed" candidate_uuid: "6914d11c690c4be79eb382496e22e2ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53076187bcd74d649c3b79fecaaf6892"
I20260925 01:26:16.304920 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f67415f49148e28b751801674bf1ed" candidate_uuid: "6914d11c690c4be79eb382496e22e2ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2979aba9db4b33aed65f8666a6e38f"
I20260925 01:26:16.305003 21384 raft_consensus.cc:3037] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:16.305106 21251 raft_consensus.cc:3037] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:16.307312 21384 raft_consensus.cc:2454] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6914d11c690c4be79eb382496e22e2ad in term 1.
I20260925 01:26:16.307463 21251 raft_consensus.cc:2454] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6914d11c690c4be79eb382496e22e2ad in term 1.
I20260925 01:26:16.307665 21051 leader_election.cc:302] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53076187bcd74d649c3b79fecaaf6892, 6914d11c690c4be79eb382496e22e2ad; no voters:
I20260925 01:26:16.307935 21456 raft_consensus.cc:2781] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:16.308334 21456 raft_consensus.cc:686] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 1 LEADER]: Becoming Leader. State: Replica: 6914d11c690c4be79eb382496e22e2ad, State: Running, Role: LEADER
I20260925 01:26:16.308761 21456 consensus_queue.cc:237] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } }
I20260925 01:26:16.313458 20972 catalog_manager.cc:5495] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 6914d11c690c4be79eb382496e22e2ad (127.6.236.129). New cstate: current_term: 1 leader_uuid: "6914d11c690c4be79eb382496e22e2ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } health_report { overall_health: HEALTHY } } }
W20260925 01:26:16.320519 21430 tablet.cc:2369] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:16.329828 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.324774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.6.236.129:33505)
0925 01:26:16.324984 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.324989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.325013 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:26:16.325209 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.325213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.325304 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.325421 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:26:16.325717 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.326456 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.327914 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.328317 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.328409 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.328841 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:26:16.329316 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:26:16.329418 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.329472 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.329535 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:26:16.337309 21384 raft_consensus.cc:1261] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Refusing update from remote peer 6914d11c690c4be79eb382496e22e2ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:16.337321 21251 raft_consensus.cc:1261] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Refusing update from remote peer 6914d11c690c4be79eb382496e22e2ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:16.338215 21459 consensus_queue.cc:1035] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:16.338608 21456 consensus_queue.cc:1035] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:26:16.344467 21297 tablet.cc:2369] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:16.349222 21462 mvcc.cc:205] Tried to move back new op lower bound from 7333067064668585984 to 7333067064564154368. Current Snapshot: MvccSnapshot[applied={T|T < 7333067064668585984}]
I20260925 01:26:16.351104 21463 mvcc.cc:205] Tried to move back new op lower bound from 7333067064668585984 to 7333067064564154368. Current Snapshot: MvccSnapshot[applied={T|T < 7333067064668585984}]
I20260925 01:26:16.361230 20972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:16.361488 20972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:16.364885 20972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Sending DeleteTablet for 3 replicas of tablet d8f67415f49148e28b751801674bf1ed
I20260925 01:26:16.366097 21364 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f67415f49148e28b751801674bf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:16 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:35616
I20260925 01:26:16.366093 21231 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f67415f49148e28b751801674bf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:16 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:57170
I20260925 01:26:16.366230 21097 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f67415f49148e28b751801674bf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:16 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:43450
I20260925 01:26:16.367394 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21034
I20260925 01:26:16.381480 21473 tablet_replica.cc:330] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: stopping tablet replica
I20260925 01:26:16.382899 21473 raft_consensus.cc:2229] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:26:16.382978 21471 tablet_replica.cc:330] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: stopping tablet replica
I20260925 01:26:16.383404 21471 raft_consensus.cc:2229] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:16.383956 21471 raft_consensus.cc:2258] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:16.384292 21472 tablet_replica.cc:330] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: stopping tablet replica
I20260925 01:26:16.384666 21473 raft_consensus.cc:2258] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:16.384757 21472 raft_consensus.cc:2229] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:16.385237 21472 raft_consensus.cc:2258] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:16.387434 21472 ts_tablet_manager.cc:1905] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:16.387439 21473 ts_tablet_manager.cc:1905] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:16.388055 21471 ts_tablet_manager.cc:1905] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:16.394109 21472 ts_tablet_manager.cc:1918] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:16.394377 21472 log.cc:1196] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/d8f67415f49148e28b751801674bf1ed
I20260925 01:26:16.394454 21471 ts_tablet_manager.cc:1918] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:16.394764 21473 ts_tablet_manager.cc:1918] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:16.394949 21472 ts_tablet_manager.cc:1939] T d8f67415f49148e28b751801674bf1ed P 2c2979aba9db4b33aed65f8666a6e38f: Deleting consensus metadata
I20260925 01:26:16.395027 21473 log.cc:1196] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/d8f67415f49148e28b751801674bf1ed
I20260925 01:26:16.395637 21473 ts_tablet_manager.cc:1939] T d8f67415f49148e28b751801674bf1ed P 6914d11c690c4be79eb382496e22e2ad: Deleting consensus metadata
I20260925 01:26:16.396886 20960 catalog_manager.cc:4841] TS 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703): tablet d8f67415f49148e28b751801674bf1ed (table test-table [id=8e94926c28df4812bc90538887ab6e02]) successfully deleted
I20260925 01:26:16.397440 20958 catalog_manager.cc:4841] TS 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505): tablet d8f67415f49148e28b751801674bf1ed (table test-table [id=8e94926c28df4812bc90538887ab6e02]) successfully deleted
I20260925 01:26:16.398304 21471 log.cc:1196] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/d8f67415f49148e28b751801674bf1ed
I20260925 01:26:16.398833 21471 ts_tablet_manager.cc:1939] T d8f67415f49148e28b751801674bf1ed P 53076187bcd74d649c3b79fecaaf6892: Deleting consensus metadata
I20260925 01:26:16.400415 20958 catalog_manager.cc:4841] TS 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263): tablet d8f67415f49148e28b751801674bf1ed (table test-table [id=8e94926c28df4812bc90538887ab6e02]) successfully deleted
Sep 25 01:26:16 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:26:16.407415 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.368604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.6.236.129:33505)
0925 01:26:16.368780 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.368786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.381836 (+ 13050us) server_negotiation.cc:407] Connection header received
0925 01:26:16.382097 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.382103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.382261 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.382373 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:16.384310 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.385490 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.395543 (+ 10053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.396265 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.396375 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.401605 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:16.401617 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:16.401630 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:16.401667 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:16.404473 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.405175 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.405182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.405184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.405333 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.405810 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.405814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.405816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.406146 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:16.406453 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.406829 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.407066 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:26:16.408001 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:16.506995 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21034
I20260925 01:26:16.522874 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21168
Sep 25 01:26:16 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:26:16.537912 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.523741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.6.236.130:36703)
0925 01:26:16.523932 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.523937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.523955 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:26:16.524013 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.524017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.524075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.524160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:16.525428 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.526186 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.527540 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.527948 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.528055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.532358 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:16.532373 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:16.532388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:16.532421 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:16.535242 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.535912 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.535920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.535922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.536040 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.536396 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.536400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.536402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.536783 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:16.537062 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.537386 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.537604 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:26:16.538682 21291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:16.626051 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21168
I20260925 01:26:16.641158 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21301
Sep 25 01:26:16 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:26:16.656883 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:16.641981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.6.236.131:46263)
0925 01:26:16.642171 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:16.642177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:16.642197 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:26:16.642325 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:16.642329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:16.642394 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:16.642480 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:16.643962 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.644695 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.646001 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:16.646573 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:16.646743 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:16.651223 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:16.651240 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:16.651256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:16.651299 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:16.653817 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.654526 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.654534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.654536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.654680 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:16.655050 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:16.655054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:16.655056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:16.655410 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:16.655691 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:16.656214 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 01:26:16.656440 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:26:16.657568 21424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:16.745226 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21301
I20260925 01:26:16.759471 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 20941
I20260925 01:26:16.760833 21003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:16.871852 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 20941
I20260925 01:26:16.889904 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--webserver_interface=127.6.236.190
--webserver_port=36915
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:17.207893 21479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:17.208168 21479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:17.208293 21479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:17.215437 21479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:17.215533 21479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:17.215591 21479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:17.215646 21479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:17.215688 21479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:17.225028 21479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=36915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:17.226222 21479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:17.227455 21479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:17.248733 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:17.248755 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:17.248718 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:17.249558 21479 server_base.cc:1034] running on GCE node
I20260925 01:26:17.250463 21479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:17.252023 21479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:17.253228 21479 hybrid_clock.cc:648] HybridClock initialized: now 1790299577253173 us; error 41 us; skew 500 ppm
I20260925 01:26:17.253619 21479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:17 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:17.256973 21479 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:17.258522 21479 webserver.cc:460] Webserver started at http://127.6.236.190:36915/ using document root <none> and password file <none>
I20260925 01:26:17.259224 21479 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:17.259341 21479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:17.265406 21479 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 01:26:17.268482 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:17.269851 21479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:26:17.270009 21479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "874e75dc42e7459aa823d025ea1d7ce8"
format_stamp: "Formatted at 2026-09-25 01:26:14 on dist-test-slave-s0b2"
server_key: "72a4d7ac722d5abf66f4e1285d27726b"
server_key_iv: "791c20ca57d4fe7e63ddf4e7e8dafc19"
server_key_version: "encryptionkey@0"
I20260925 01:26:17.271162 21479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:17.299296 21479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:17.303411 21479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:17.303675 21479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:17.320854 21479 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46041
I20260925 01:26:17.320875 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46041 every 8 connection(s)
I20260925 01:26:17.322364 21479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:17.329249 21547 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:26:17.331655 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21479
I20260925 01:26:17.332425 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:33505
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=35023
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:17.332513 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap starting.
I20260925 01:26:17.351263 21547 log.cc:824] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:17.363126 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:26:17.363613 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap complete.
I20260925 01:26:17.372448 21547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:17.373273 21547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Initialized, Role: FOLLOWER
I20260925 01:26:17.373870 21547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:17.374094 21547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:17.374212 21547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:17.374415 21547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:26:17.378209 21547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:17.378741 21547 leader_election.cc:302] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874e75dc42e7459aa823d025ea1d7ce8; no voters:
I20260925 01:26:17.379300 21547 leader_election.cc:288] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:26:17.379442 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:26:17.380579 21551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 LEADER]: Becoming Leader. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Running, Role: LEADER
I20260925 01:26:17.381191 21551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:17.381577 21547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:17.383752 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874e75dc42e7459aa823d025ea1d7ce8. Latest consensus state: current_term: 2 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:17.383805 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:17.384114 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:17.384162 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:17.386602 21558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:17.402486 21558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:17.404067 21558 catalog_manager.cc:1259] Loaded cluster ID: 6f403c2cc004401182e929b4793fc0d8
I20260925 01:26:17.404246 21558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:17.407627 21558 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 01:26:17.426738 21558 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 01:26:17.446328 21558 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:17.447700 21558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Loaded TSK: 0
I20260925 01:26:17.448734 21558 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:26:17.698561 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:17.698891 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:17.699041 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:17.706411 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:17.706511 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:17.706663 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:26:17.716318 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:33505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=35023
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:17.717569 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:17.719094 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:17.733711 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:17.733628 21575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:17.733862 21577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:17.734736 21549 server_base.cc:1034] running on GCE node
I20260925 01:26:17.735317 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:17.736270 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:17.737469 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790299577737400 us; error 61 us; skew 500 ppm
I20260925 01:26:17.737785 21549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:17 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:17.740901 21549 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:26:17.742399 21549 webserver.cc:460] Webserver started at http://127.6.236.129:35023/ using document root <none> and password file <none>
I20260925 01:26:17.743073 21549 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:17.743194 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:17.748461 21549 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 01:26:17.751169 21584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:17.752485 21549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:26:17.752636 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "6914d11c690c4be79eb382496e22e2ad"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "26b9e597080d02e746e02f59c9dbcd7b"
server_key_iv: "e418c9e235d277beb10ac1e1f5997662"
server_key_version: "encryptionkey@0"
I20260925 01:26:17.753547 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:17.778122 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:17.782200 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:17.782553 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:17.783699 21549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:17.784955 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:17.785056 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:17.785161 21549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:17.785228 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:17.815361 21549 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:33505
I20260925 01:26:17.815423 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:33505 every 8 connection(s)
I20260925 01:26:17.816762 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:26:17.823740 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21549
I20260925 01:26:17.824371 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:36703
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=44949
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:26:17 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:17.840850 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:17.819511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:38605 (local address 127.6.236.190:46041)
0925 01:26:17.820069 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:17.820083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:17.820631 (+ 548us) server_negotiation.cc:407] Connection header received
0925 01:26:17.822749 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:17.822763 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:17.823112 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:17.823532 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:17.825100 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:17.827063 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:17.829038 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:17.829956 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:17.830122 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:17.833894 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:17.833902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:17.833916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:17.833953 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:17.836664 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:17.837498 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:17.837504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:17.837510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:17.837703 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:17.838103 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:17.838105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:17.838107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:17.838600 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:17.838781 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:17.839585 (+ 804us) server_negotiation.cc:299] Negotiation successful
0925 01:26:17.839922 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
I20260925 01:26:17.841080 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:17.819653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:38605)
0925 01:26:17.820445 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:17.820497 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:17.822195 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:17.823677 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:17.823685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:17.824243 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:17.824860 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:17.824879 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:17.827305 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:17.827309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:17.828887 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:17.828902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:17.830162 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:17.830168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:17.830431 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:17.831113 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:17.831140 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:17.833618 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:17.836805 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:17.836814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:17.836829 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:17.837384 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:17.837796 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:17.837801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:17.837803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:17.838015 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:17.838757 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:17.838765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:17.839134 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 01:26:17.839402 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 01:26:17.839929 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":331,"threads_started":1}
I20260925 01:26:17.844177 21697 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:17.844516 21697 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:17.845422 21697 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
I20260925 01:26:17.847754 21512 ts_manager.cc:194] Registered new tserver with Master: 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505)
I20260925 01:26:17.849263 21512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:38605
W20260925 01:26:18.136816 21701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:18.137068 21701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:18.137178 21701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:18.144294 21701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:18.144397 21701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:18.144538 21701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:18.154026 21701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:36703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=44949
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:18.155304 21701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:18.156621 21701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:18.168865 21709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:18.168843 21707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:18.169334 21706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:18.169504 21701 server_base.cc:1034] running on GCE node
I20260925 01:26:18.170089 21701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:18.170977 21701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:18.172144 21701 hybrid_clock.cc:648] HybridClock initialized: now 1790299578172093 us; error 48 us; skew 500 ppm
I20260925 01:26:18.172410 21701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:18 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:18.175462 21701 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:18.176832 21701 webserver.cc:460] Webserver started at http://127.6.236.130:44949/ using document root <none> and password file <none>
I20260925 01:26:18.177479 21701 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:18.177599 21701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:18.182739 21701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:18.185256 21716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.186616 21701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:18.186774 21701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "2c2979aba9db4b33aed65f8666a6e38f"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "c81912d71b592c28994542b6fa3a6d7f"
server_key_iv: "b960f38ce3e728cd0fe714e1edbdf9d3"
server_key_version: "encryptionkey@0"
I20260925 01:26:18.187722 21701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:18.215904 21701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:18.219894 21701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:18.220132 21701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:18.221066 21701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:18.222362 21701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:18.222455 21701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.222559 21701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:18.222630 21701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.252179 21701 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36703
I20260925 01:26:18.252247 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36703 every 8 connection(s)
I20260925 01:26:18.253551 21701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:18.258379 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21701
I20260925 01:26:18.259097 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:46263
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=38743
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:26:18 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:18.272944 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.255938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:39623 (local address 127.6.236.190:46041)
0925 01:26:18.256118 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.256123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.257302 (+ 1179us) server_negotiation.cc:407] Connection header received
0925 01:26:18.259676 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.259680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.259755 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.259842 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:18.261738 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.262462 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.263986 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.264405 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.264520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.267708 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:18.267713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:18.267716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:18.267748 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:18.270088 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.270773 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.270778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.270781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.270906 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.271307 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.271310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.271312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.271580 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:18.271714 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.272389 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.272620 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:26:18.273900 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.256465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:39623)
0925 01:26:18.257142 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.257185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.259380 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.260099 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.260111 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.260809 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:18.261563 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.261597 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.262611 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.262614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.263840 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.263852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.264618 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.264625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.264851 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.265449 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:18.265471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:18.267509 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:18.270217 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:18.270226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:18.270261 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:18.270658 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:18.271006 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:18.271011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:18.271015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:18.271192 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:18.271710 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:18.271718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:18.272030 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.272259 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.272778 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":241,"threads_started":1}
I20260925 01:26:18.275316 21829 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:18.275590 21829 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:18.276271 21829 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
I20260925 01:26:18.277693 21512 ts_manager.cc:194] Registered new tserver with Master: 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:18.278852 21512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:39623
W20260925 01:26:18.572734 21832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:18.573009 21832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:18.573109 21832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:18.580449 21832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:18.580564 21832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:18.580690 21832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:26:18.590385 21832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:46263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=38743
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:18.591806 21832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:18.593070 21832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:18.605286 21840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:18.605222 21838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:18.605623 21837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:18.606132 21832 server_base.cc:1034] running on GCE node
I20260925 01:26:18.606774 21832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:18.607662 21832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:18.608884 21832 hybrid_clock.cc:648] HybridClock initialized: now 1790299578608826 us; error 38 us; skew 500 ppm
I20260925 01:26:18.609212 21832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:18 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:18.612390 21832 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:18.613832 21832 webserver.cc:460] Webserver started at http://127.6.236.131:38743/ using document root <none> and password file <none>
I20260925 01:26:18.614557 21832 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:18.614691 21832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:18.620067 21832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:26:18.622722 21847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.623998 21832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:26:18.624150 21832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "53076187bcd74d649c3b79fecaaf6892"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "1fa3a9fdbedc1ae3e3a6bb97fcafcb1f"
server_key_iv: "384e511e9534ca2a15be5e4dda083336"
server_key_version: "encryptionkey@0"
I20260925 01:26:18.625118 21832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:18.657428 21832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:18.661679 21832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:18.661931 21832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:18.662962 21832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:18.664297 21832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:18.664395 21832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.664508 21832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:18.664587 21832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:18.694144 21832 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:46263
I20260925 01:26:18.694209 21959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:46263 every 8 connection(s)
I20260925 01:26:18.695616 21832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:18.700870 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 21832
Sep 25 01:26:18 dist-test-slave-s0b2 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:18.717741 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.698123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:44761 (local address 127.6.236.190:46041)
0925 01:26:18.698341 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.698347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.699604 (+ 1257us) server_negotiation.cc:407] Connection header received
0925 01:26:18.701789 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.701795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.701885 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.701971 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:18.703487 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.704203 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.706535 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.707189 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.707322 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.711635 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:18.711641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:18.711647 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:18.711684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:18.714821 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.715515 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.715520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.715525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.715658 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.716060 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.716065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.716069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.716332 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:18.716463 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.717151 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.717391 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:26:18.718801 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.698650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:44761)
0925 01:26:18.699417 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.699474 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.701491 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.702219 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.702228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.702775 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:18.703324 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.703341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.704364 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.704370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.706327 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.706343 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.707656 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.707666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.707962 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.708654 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:18.708685 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:18.711409 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:18.714952 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:18.714961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:18.714979 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:18.715394 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:18.715757 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:18.715762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:18.715766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:18.715946 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:18.716451 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:18.716459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:18.716790 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.717024 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.717545 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":296,"threads_started":1}
I20260925 01:26:18.720198 21960 heartbeater.cc:347] Connected to a master server at 127.6.236.190:46041
I20260925 01:26:18.720507 21960 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:18.721343 21960 heartbeater.cc:511] Master 127.6.236.190:46041 requested a full tablet report, sending...
I20260925 01:26:18.721688 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.701937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.6.236.190:46041)
0925 01:26:18.704941 (+ 3004us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.704946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.704960 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:26:18.705001 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.705004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.705058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.705149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:18.707350 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.708487 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.712973 (+ 4486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.713970 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.714099 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.716639 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:18.716643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:18.716647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:18.716674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:18.719285 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.719908 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.719911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.719913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.720022 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:18.720402 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:18.720406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:18.720408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:18.720689 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:18.720879 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.721216 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.721375 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2919,"thread_start_us":2802,"threads_started":1}
I20260925 01:26:18.722934 21512 ts_manager.cc:194] Registered new tserver with Master: 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263)
I20260925 01:26:18.723779 21512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:44761
I20260925 01:26:18.724360 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:26:18.731009 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.725288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.6.236.190:46041)
0925 01:26:18.725454 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.725459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.725930 (+ 471us) server_negotiation.cc:407] Connection header received
0925 01:26:18.726090 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.726104 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.726309 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.726436 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:26:18.726816 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.727525 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.728958 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.729497 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.729612 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.730067 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:26:18.730587 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:26:18.730697 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.730770 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.730833 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:26:18.732088 21512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:26:18.739392 21512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:26:18.760388 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.750878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36703 (local address 127.0.0.1:57182)
0925 01:26:18.752966 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.752985 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.753112 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.753944 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.753948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.753972 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:18.754322 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.754335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.755881 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.755886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.757918 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.757931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.759440 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.759447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.759650 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.759688 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.759808 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.759991 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969,"mutex_wait_us":41,"thread_start_us":1839,"threads_started":1}
I20260925 01:26:18.760375 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.751561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.6.236.130:36703)
0925 01:26:18.752556 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.752571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.753249 (+ 678us) server_negotiation.cc:407] Connection header received
0925 01:26:18.753340 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.753348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.753560 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.753743 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:18.754493 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.755725 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.758100 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.759222 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.759342 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.759572 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.759873 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.760032 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":699,"threads_started":1}
I20260925 01:26:18.760648 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.750821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.6.236.129:33505)
0925 01:26:18.751413 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.751423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.752131 (+ 708us) server_negotiation.cc:407] Connection header received
0925 01:26:18.752527 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.752535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.752811 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.752973 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:18.753934 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.755455 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.757841 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.759291 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.759390 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.759614 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.760191 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.760357 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":1141,"threads_started":1}
I20260925 01:26:18.761412 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.750585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:33505 (local address 127.0.0.1:43470)
0925 01:26:18.751991 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.752031 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.752283 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.753270 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.753277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.753306 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:18.753704 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.753718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.755589 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.755594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.757656 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.757671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.759807 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.759812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.759974 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.760005 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.760873 (+ 868us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.761006 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":67,"thread_start_us":1034,"threads_started":1}
I20260925 01:26:18.762188 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.752435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:46263 (local address 127.0.0.1:35630)
0925 01:26:18.753257 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.753275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.753392 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.754511 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.754516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.754541 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:18.754919 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.754932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.757020 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.757026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.759213 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.759228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.761435 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.761442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.761630 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.761662 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.761763 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.761882 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":863,"threads_started":1}
I20260925 01:26:18.762418 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.753036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.6.236.131:46263)
0925 01:26:18.753932 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.753941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.753970 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:26:18.754064 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.754070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.754312 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.755147 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:18.755375 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.756877 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.759401 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.761197 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.761331 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.761716 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.761796 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.761942 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":596,"threads_started":1}
I20260925 01:26:18.764002 21764 tablet_service.cc:1467] Processing CreateTablet for tablet 74785e530d12409aa3964b018944a666 (DEFAULT_TABLE table=test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:18.765668 21764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74785e530d12409aa3964b018944a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:18.768234 21895 tablet_service.cc:1467] Processing CreateTablet for tablet 74785e530d12409aa3964b018944a666 (DEFAULT_TABLE table=test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:18.768234 21632 tablet_service.cc:1467] Processing CreateTablet for tablet 74785e530d12409aa3964b018944a666 (DEFAULT_TABLE table=test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:18.769562 21895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74785e530d12409aa3964b018944a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:18.769567 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74785e530d12409aa3964b018944a666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:18.777760 21976 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Bootstrap starting.
I20260925 01:26:18.780938 21976 tablet_bootstrap.cc:654] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:18.781972 21976 log.cc:824] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:18.784196 21979 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Bootstrap starting.
I20260925 01:26:18.784752 21976 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: No bootstrap required, opened a new log
I20260925 01:26:18.785032 21976 ts_tablet_manager.cc:1397] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 01:26:18.786548 21978 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Bootstrap starting.
I20260925 01:26:18.789042 21979 tablet_bootstrap.cc:654] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:18.789397 21978 tablet_bootstrap.cc:654] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:18.790349 21979 log.cc:824] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:18.790634 21978 log.cc:824] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:18.793386 21979 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: No bootstrap required, opened a new log
I20260925 01:26:18.793660 21979 ts_tablet_manager.cc:1397] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 01:26:18.795058 21978 tablet_bootstrap.cc:492] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: No bootstrap required, opened a new log
I20260925 01:26:18.795287 21978 ts_tablet_manager.cc:1397] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 01:26:18.795001 21976 raft_consensus.cc:348] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.795411 21976 raft_consensus.cc:374] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:18.795502 21976 raft_consensus.cc:729] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2979aba9db4b33aed65f8666a6e38f, State: Initialized, Role: FOLLOWER
I20260925 01:26:18.796108 21976 consensus_queue.cc:260] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.797178 21829 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:18.797850 21976 ts_tablet_manager.cc:1428] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260925 01:26:18.803503 21979 raft_consensus.cc:348] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.803843 21979 raft_consensus.cc:374] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:18.803932 21979 raft_consensus.cc:729] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6914d11c690c4be79eb382496e22e2ad, State: Initialized, Role: FOLLOWER
I20260925 01:26:18.804276 21978 raft_consensus.cc:348] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.804431 21979 consensus_queue.cc:260] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.804592 21978 raft_consensus.cc:374] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:18.804672 21978 raft_consensus.cc:729] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53076187bcd74d649c3b79fecaaf6892, State: Initialized, Role: FOLLOWER
I20260925 01:26:18.805213 21978 consensus_queue.cc:260] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.805444 21697 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:18.806228 21960 heartbeater.cc:503] Master 127.6.236.190:46041 was elected leader, sending a full tablet report...
I20260925 01:26:18.806839 21978 ts_tablet_manager.cc:1428] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 01:26:18.807039 21979 ts_tablet_manager.cc:1428] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260925 01:26:18.818818 21698 tablet.cc:2369] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:18.934125 21984 raft_consensus.cc:482] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:18.934453 21984 raft_consensus.cc:504] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.936267 21984 leader_election.cc:288] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505), 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:18.943347 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.936716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:33747 (local address 127.6.236.130:36703)
0925 01:26:18.936914 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.936920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.937037 (+ 117us) server_negotiation.cc:407] Connection header received
0925 01:26:18.937282 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.937288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.937368 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.937486 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:18.937982 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.938966 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.941409 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.942606 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.942770 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.942842 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.943034 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.943139 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:26:18.943670 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.936802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:46567 (local address 127.6.236.129:33505)
0925 01:26:18.936985 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.936991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.937548 (+ 557us) server_negotiation.cc:407] Connection header received
0925 01:26:18.937662 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.937666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.937737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.937830 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:18.938478 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.939321 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.941637 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.942836 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.942960 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.943025 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.943328 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.943454 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:26:18.943672 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.936604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36703 (local address 127.6.236.131:33747)
0925 01:26:18.936924 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.936944 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.937112 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.937546 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.937551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.937592 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:18.937828 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.937840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.939097 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.939101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.941235 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.941251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.942746 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.942750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.942876 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.942906 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.943009 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.943146 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":82}
I20260925 01:26:18.944010 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.936692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:33505 (local address 127.6.236.131:46567)
0925 01:26:18.937412 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:18.937433 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:18.937550 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:18.937862 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:18.937865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:18.937885 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:18.938285 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.938298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.939471 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.939476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.941455 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:18.941466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.943012 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.943018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:18.943178 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.943208 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:26:18.943295 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 01:26:18.943401 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":23,"thread_start_us":380,"threads_started":1}
I20260925 01:26:18.944644 21784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74785e530d12409aa3964b018944a666" candidate_uuid: "53076187bcd74d649c3b79fecaaf6892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2979aba9db4b33aed65f8666a6e38f" is_pre_election: true
I20260925 01:26:18.944983 21652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74785e530d12409aa3964b018944a666" candidate_uuid: "53076187bcd74d649c3b79fecaaf6892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6914d11c690c4be79eb382496e22e2ad" is_pre_election: true
I20260925 01:26:18.945214 21784 raft_consensus.cc:2454] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53076187bcd74d649c3b79fecaaf6892 in term 0.
I20260925 01:26:18.945531 21652 raft_consensus.cc:2454] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53076187bcd74d649c3b79fecaaf6892 in term 0.
I20260925 01:26:18.945912 21851 leader_election.cc:302] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2979aba9db4b33aed65f8666a6e38f, 53076187bcd74d649c3b79fecaaf6892; no voters:
I20260925 01:26:18.946319 21984 raft_consensus.cc:2781] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:18.946450 21984 raft_consensus.cc:482] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:18.946548 21984 raft_consensus.cc:3037] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 0 FOLLOWER]: Advancing to term 1
W20260925 01:26:18.947034 21961 tablet.cc:2369] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:18.948760 21984 raft_consensus.cc:504] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.949368 21984 leader_election.cc:288] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [CANDIDATE]: Term 1 election: Requested vote from peers 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505), 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703)
I20260925 01:26:18.949725 21652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74785e530d12409aa3964b018944a666" candidate_uuid: "53076187bcd74d649c3b79fecaaf6892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6914d11c690c4be79eb382496e22e2ad"
I20260925 01:26:18.949887 21784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74785e530d12409aa3964b018944a666" candidate_uuid: "53076187bcd74d649c3b79fecaaf6892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2979aba9db4b33aed65f8666a6e38f"
I20260925 01:26:18.949927 21652 raft_consensus.cc:3037] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:18.950086 21784 raft_consensus.cc:3037] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:18.953020 21652 raft_consensus.cc:2454] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53076187bcd74d649c3b79fecaaf6892 in term 1.
I20260925 01:26:18.953084 21784 raft_consensus.cc:2454] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53076187bcd74d649c3b79fecaaf6892 in term 1.
I20260925 01:26:18.953433 21851 leader_election.cc:302] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c2979aba9db4b33aed65f8666a6e38f, 53076187bcd74d649c3b79fecaaf6892, 6914d11c690c4be79eb382496e22e2ad; no voters:
I20260925 01:26:18.953744 21984 raft_consensus.cc:2781] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:18.954161 21984 raft_consensus.cc:686] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 1 LEADER]: Becoming Leader. State: Replica: 53076187bcd74d649c3b79fecaaf6892, State: Running, Role: LEADER
I20260925 01:26:18.954625 21984 consensus_queue.cc:237] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } }
I20260925 01:26:18.959159 21512 catalog_manager.cc:5495] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "53076187bcd74d649c3b79fecaaf6892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53076187bcd74d649c3b79fecaaf6892" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 46263 } health_report { overall_health: HEALTHY } } }
I20260925 01:26:18.996763 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:18.991886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.6.236.131:46263)
0925 01:26:18.992051 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:18.992078 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:18.992107 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:26:18.992215 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:18.992220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:18.992323 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:18.992473 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:26:18.992844 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.993560 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.994931 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:18.995325 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:18.995415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:18.995869 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:26:18.996357 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:26:18.996467 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:18.996503 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 01:26:18.996565 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:26:19.004746 21652 raft_consensus.cc:1261] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Refusing update from remote peer 53076187bcd74d649c3b79fecaaf6892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:19.004741 21784 raft_consensus.cc:1261] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Refusing update from remote peer 53076187bcd74d649c3b79fecaaf6892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 01:26:19.005424 21830 tablet.cc:2369] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:19.005752 21986 consensus_queue.cc:1035] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6914d11c690c4be79eb382496e22e2ad" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:19.006083 21984 consensus_queue.cc:1035] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2979aba9db4b33aed65f8666a6e38f" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:19.029245 21512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:19.029472 21512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:19.032897 21512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Sending DeleteTablet for 3 replicas of tablet 74785e530d12409aa3964b018944a666
I20260925 01:26:19.033923 21764 tablet_service.cc:1514] Processing DeleteTablet for tablet 74785e530d12409aa3964b018944a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:19 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:57182
I20260925 01:26:19.033991 21895 tablet_service.cc:1514] Processing DeleteTablet for tablet 74785e530d12409aa3964b018944a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:19 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:35630
I20260925 01:26:19.034498 21632 tablet_service.cc:1514] Processing DeleteTablet for tablet 74785e530d12409aa3964b018944a666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:19 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:43470
I20260925 01:26:19.035436 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21549
I20260925 01:26:19.045084 22001 tablet_replica.cc:330] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: stopping tablet replica
I20260925 01:26:19.045656 22001 raft_consensus.cc:2229] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:19.046307 22001 raft_consensus.cc:2258] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:19.048558 22001 ts_tablet_manager.cc:1905] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:19.055820 22001 ts_tablet_manager.cc:1918] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:19.056851 21999 tablet_replica.cc:330] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: stopping tablet replica
I20260925 01:26:19.057168 22000 tablet_replica.cc:330] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: stopping tablet replica
I20260925 01:26:19.057898 21999 raft_consensus.cc:2229] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:26:19.058393 22000 raft_consensus.cc:2229] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:19.058936 22000 raft_consensus.cc:2258] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:19.059841 21999 raft_consensus.cc:2258] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:19.060240 22001 log.cc:1196] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/74785e530d12409aa3964b018944a666
I20260925 01:26:19.060791 22001 ts_tablet_manager.cc:1939] T 74785e530d12409aa3964b018944a666 P 6914d11c690c4be79eb382496e22e2ad: Deleting consensus metadata
I20260925 01:26:19.061084 22000 ts_tablet_manager.cc:1905] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:19.062333 21999 ts_tablet_manager.cc:1905] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:19.062701 21497 catalog_manager.cc:4841] TS 6914d11c690c4be79eb382496e22e2ad (127.6.236.129:33505): tablet 74785e530d12409aa3964b018944a666 (table test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]) successfully deleted
I20260925 01:26:19.058327 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:19.036206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.6.236.129:33505)
0925 01:26:19.036373 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:19.036379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:19.036733 (+ 354us) server_negotiation.cc:407] Connection header received
0925 01:26:19.036801 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:19.036807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:19.036888 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:19.036975 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:19.038998 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.040242 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.042275 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.042984 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.043110 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:19.049469 (+ 6359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:19.049488 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:19.049507 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:19.049557 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:19.053665 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.054694 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.054705 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.054709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.054935 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.056454 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.056460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.056463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.056860 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:19.057192 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:19.057571 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 01:26:19.057790 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:26:19.066917 21692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:19.067972 22000 ts_tablet_manager.cc:1918] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:19.068305 22000 log.cc:1196] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/74785e530d12409aa3964b018944a666
I20260925 01:26:19.068931 22000 ts_tablet_manager.cc:1939] T 74785e530d12409aa3964b018944a666 P 2c2979aba9db4b33aed65f8666a6e38f: Deleting consensus metadata
I20260925 01:26:19.070914 21499 catalog_manager.cc:4841] TS 2c2979aba9db4b33aed65f8666a6e38f (127.6.236.130:36703): tablet 74785e530d12409aa3964b018944a666 (table test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]) successfully deleted
I20260925 01:26:19.071184 21999 ts_tablet_manager.cc:1918] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:19.071578 21999 log.cc:1196] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/74785e530d12409aa3964b018944a666
I20260925 01:26:19.072140 21999 ts_tablet_manager.cc:1939] T 74785e530d12409aa3964b018944a666 P 53076187bcd74d649c3b79fecaaf6892: Deleting consensus metadata
I20260925 01:26:19.074512 21497 catalog_manager.cc:4841] TS 53076187bcd74d649c3b79fecaaf6892 (127.6.236.131:46263): tablet 74785e530d12409aa3964b018944a666 (table test-table [id=9688b7cd9e6c4141a815f0aa7641eee6]) successfully deleted
I20260925 01:26:19.161406 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21549
I20260925 01:26:19.183365 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21701
I20260925 01:26:19.200479 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:19.184313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.6.236.130:36703)
0925 01:26:19.184472 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:19.184477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:19.184495 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:26:19.184633 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:19.184636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:19.184698 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:19.184787 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:19.186620 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.187803 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.190142 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.190837 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.190969 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:19.194849 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:19.194861 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:19.194877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:19.194911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:19.197696 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.198543 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.198581 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.198583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.198706 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.199074 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.199078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.199081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.199442 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:19.199683 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:19.200032 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 01:26:19.200197 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:26:19.201159 21824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:19.288612 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21701
I20260925 01:26:19.303567 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21832
I20260925 01:26:19.320533 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:19.304313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.6.236.131:46263)
0925 01:26:19.304469 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:19.304473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:19.304532 (+ 59us) server_negotiation.cc:407] Connection header received
0925 01:26:19.304730 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:19.304734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:19.304794 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:19.304880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:19.306841 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.307777 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.309677 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:19.310115 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:19.310212 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:19.313785 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:19.313798 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:19.313819 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:19.313861 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:19.316843 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.317725 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.317732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.317735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.317890 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:19.318474 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:19.318479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:19.318482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:19.318980 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:19.319270 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:19.319913 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 01:26:19.320092 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:26:19.321367 21955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:19.445564 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21832
I20260925 01:26:19.462293 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 21479
I20260925 01:26:19.463374 21542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:19.575268 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 21479
I20260925 01:26:19.593608 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--webserver_interface=127.6.236.190
--webserver_port=36915
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:19.906318 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:19.906587 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:19.906708 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:19.913692 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:19.913789 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:19.913857 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:19.913931 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:19.913986 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:19.923709 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=36915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:19.924942 22006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:19.926255 22006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:19.937315 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:19.937356 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:19.937397 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:19.938227 22006 server_base.cc:1034] running on GCE node
I20260925 01:26:19.938944 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:19.940231 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:19.941435 22006 hybrid_clock.cc:648] HybridClock initialized: now 1790299579941389 us; error 28 us; skew 500 ppm
I20260925 01:26:19.941763 22006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:19 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299579, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:19.949307 22006 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:19.950677 22006 webserver.cc:460] Webserver started at http://127.6.236.190:36915/ using document root <none> and password file <none>
I20260925 01:26:19.951316 22006 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:19.951432 22006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:19.956537 22006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:26:19.959177 22021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:19.960362 22006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:19.960511 22006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "874e75dc42e7459aa823d025ea1d7ce8"
format_stamp: "Formatted at 2026-09-25 01:26:14 on dist-test-slave-s0b2"
server_key: "72a4d7ac722d5abf66f4e1285d27726b"
server_key_iv: "791c20ca57d4fe7e63ddf4e7e8dafc19"
server_key_version: "encryptionkey@0"
I20260925 01:26:19.961431 22006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:19.995113 22006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:19.999258 22006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:19.999495 22006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:20.016201 22006 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46041
I20260925 01:26:20.016230 22072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46041 every 8 connection(s)
I20260925 01:26:20.017617 22006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:20.024209 22073 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:26:20.024348 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22006
I20260925 01:26:20.025151 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:33505
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=35023
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:20.027477 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap starting.
I20260925 01:26:20.053465 22073 log.cc:824] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:20.074095 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:26:20.074721 22073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8: Bootstrap complete.
I20260925 01:26:20.087740 22073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:20.089079 22073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Initialized, Role: FOLLOWER
I20260925 01:26:20.089852 22073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:20.090159 22073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:20.090320 22073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:20.090530 22073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:26:20.096043 22073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:20.096621 22073 leader_election.cc:302] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874e75dc42e7459aa823d025ea1d7ce8; no voters:
I20260925 01:26:20.097283 22073 leader_election.cc:288] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:26:20.097405 22077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:26:20.098541 22077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [term 3 LEADER]: Becoming Leader. State: Replica: 874e75dc42e7459aa823d025ea1d7ce8, State: Running, Role: LEADER
I20260925 01:26:20.099071 22077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } }
I20260925 01:26:20.099375 22073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:20.101438 22079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874e75dc42e7459aa823d025ea1d7ce8. Latest consensus state: current_term: 3 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:20.101432 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "874e75dc42e7459aa823d025ea1d7ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874e75dc42e7459aa823d025ea1d7ce8" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46041 } } }
I20260925 01:26:20.101903 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:20.101980 22079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874e75dc42e7459aa823d025ea1d7ce8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:20.102825 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:20.120877 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:20.122408 22083 catalog_manager.cc:1259] Loaded cluster ID: 6f403c2cc004401182e929b4793fc0d8
I20260925 01:26:20.122586 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 01:26:20.124598 22083 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5d1fbc09ad google::LogMessage::Fail() at ??:0
@ 0x7f5d1fbc388a google::LogMessage::SendToLog() at ??:0
@ 0x7f5d1fbc04de google::LogMessage::Flush() at ??:0
@ 0x7f5d1fbc1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5d355f3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5d355f0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5d35678610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5d35678439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5d2280c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5d2280f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5d2280f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5d227eaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5d25ac46db start_thread at ??:0
@ 0x7f5d1ea5a61f clone at ??:0
W20260925 01:26:20.422856 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:20.423233 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:20.423396 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:20.433223 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:20.433324 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:20.433485 22075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:26:20.445469 22075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:33505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=35023
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:20.446854 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:20.448241 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:20.460979 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:20.460968 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:20.461006 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:20.462181 22075 server_base.cc:1034] running on GCE node
I20260925 01:26:20.462841 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:20.463773 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:20.464960 22075 hybrid_clock.cc:648] HybridClock initialized: now 1790299580464912 us; error 39 us; skew 500 ppm
I20260925 01:26:20.465303 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:20 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:20.468843 22075 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:26:20.470873 22075 webserver.cc:460] Webserver started at http://127.6.236.129:35023/ using document root <none> and password file <none>
I20260925 01:26:20.471802 22075 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:20.471983 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:20.478983 22075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:26:20.482383 22125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.484015 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:26:20.484184 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "6914d11c690c4be79eb382496e22e2ad"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "26b9e597080d02e746e02f59c9dbcd7b"
server_key_iv: "e418c9e235d277beb10ac1e1f5997662"
server_key_version: "encryptionkey@0"
I20260925 01:26:20.485517 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:20.500037 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:20.504565 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:20.504858 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:20.505960 22075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:20.507571 22075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:20.507706 22075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.507819 22075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:20.507893 22075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.539196 22075 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:33505
I20260925 01:26:20.539227 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:33505 every 8 connection(s)
I20260925 01:26:20.540812 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:26:20.542073 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22075
I20260925 01:26:20.542884 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:36703
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=44949
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:20.560675 22240 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:26:20.544233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:60533)
0925 01:26:20.545032 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.545080 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:20.546907 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:20.559235 (+ 12328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
W20260925 01:26:20.561599 22238 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:46041 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:46041: Client connection negotiation failed: client connection to 127.6.236.190:46041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:26:20.562681 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:20.562211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:49779)
0925 01:26:20.562413 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.562548 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:26:20.563773 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:20.563367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:58169)
0925 01:26:20.563524 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.563609 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:26:20.564049 22238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:26:20.858776 22241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:20.859054 22241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:20.859169 22241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:20.866317 22241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:20.866420 22241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:20.866541 22241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:20.876091 22241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:36703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=44949
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:20.877336 22241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:20.878643 22241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:20.890872 22246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:20.890942 22247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:20.890978 22249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:20.892047 22241 server_base.cc:1034] running on GCE node
I20260925 01:26:20.892633 22241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:20.893574 22241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:20.894754 22241 hybrid_clock.cc:648] HybridClock initialized: now 1790299580894698 us; error 46 us; skew 500 ppm
I20260925 01:26:20.895032 22241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:20 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:20.898017 22241 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:20.899515 22241 webserver.cc:460] Webserver started at http://127.6.236.130:44949/ using document root <none> and password file <none>
I20260925 01:26:20.900199 22241 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:20.900314 22241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:20.905561 22241 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:26:20.908249 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.909461 22241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:26:20.909636 22241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "2c2979aba9db4b33aed65f8666a6e38f"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "c81912d71b592c28994542b6fa3a6d7f"
server_key_iv: "b960f38ce3e728cd0fe714e1edbdf9d3"
server_key_version: "encryptionkey@0"
I20260925 01:26:20.910617 22241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:20.927119 22241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:20.930815 22241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:20.931068 22241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:20.932005 22241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:20.933235 22241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:20.933339 22241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.933446 22241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:20.933514 22241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:20.963130 22241 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36703
I20260925 01:26:20.963186 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36703 every 8 connection(s)
I20260925 01:26:20.964540 22241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:20.966006 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22241
I20260925 01:26:20.966742 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:46263
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:46041
--webserver_port=38743
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:26:20.972013 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:20.967413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:51951)
0925 01:26:20.970823 (+ 3410us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.971109 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3115,"thread_start_us":2933,"threads_started":1}
W20260925 01:26:20.972746 22369 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:46041 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:46041: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
I20260925 01:26:20.973608 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:20.973272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:36213)
0925 01:26:20.973427 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.973511 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 01:26:20.974422 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:20.974082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:43227)
0925 01:26:20.974217 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:20.974323 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 01:26:20.974639 22369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:26:21.277731 22372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:21.278017 22372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:21.278165 22372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:21.285152 22372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:21.285265 22372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:21.285418 22372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:26:21.295248 22372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:46263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=38743
--tserver_master_addrs=127.6.236.190:46041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:21.296476 22372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:21.297758 22372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:21.310052 22380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:21.310101 22378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:21.310063 22377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:21.311339 22372 server_base.cc:1034] running on GCE node
I20260925 01:26:21.311872 22372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:21.312733 22372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:21.313915 22372 hybrid_clock.cc:648] HybridClock initialized: now 1790299581313876 us; error 32 us; skew 500 ppm
I20260925 01:26:21.314205 22372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:21 dist-test-slave-s0b2 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:21.317138 22372 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:21.318531 22372 webserver.cc:460] Webserver started at http://127.6.236.131:38743/ using document root <none> and password file <none>
I20260925 01:26:21.319188 22372 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:21.319299 22372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:21.324473 22372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 01:26:21.327138 22387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:21.328477 22372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:26:21.328625 22372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "53076187bcd74d649c3b79fecaaf6892"
format_stamp: "Formatted at 2026-09-25 01:26:15 on dist-test-slave-s0b2"
server_key: "1fa3a9fdbedc1ae3e3a6bb97fcafcb1f"
server_key_iv: "384e511e9534ca2a15be5e4dda083336"
server_key_version: "encryptionkey@0"
I20260925 01:26:21.329576 22372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:21.343757 22372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:21.347302 22372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:21.347534 22372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:21.348423 22372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:21.349646 22372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:21.349737 22372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:21.349841 22372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:21.349913 22372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:21.379163 22372 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:46263
I20260925 01:26:21.379227 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:46263 every 8 connection(s)
I20260925 01:26:21.380613 22372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:21.385314 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:21.383547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:41495)
0925 01:26:21.384248 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:21.384475 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":255,"threads_started":1}
W20260925 01:26:21.385907 22500 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:46041 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:46041: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
I20260925 01:26:21.386610 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22372
I20260925 01:26:21.386750 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:21.386447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:58675)
0925 01:26:21.386575 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:21.386646 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 01:26:21.387538 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:21.387240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:53635)
0925 01:26:21.387366 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:21.387436 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 01:26:21.387743 22500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:26:21.565366 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:21.564599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:40335)
0925 01:26:21.565023 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:21.565116 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":209,"threads_started":1}
I20260925 01:26:21.975945 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:21.975182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:43027)
0925 01:26:21.975628 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:21.975734 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":215,"threads_started":1}
I20260925 01:26:22.388983 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:22.388240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:55581)
0925 01:26:22.388674 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:22.388774 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260925 01:26:22.566951 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:22.566219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:34095)
0925 01:26:22.566691 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:22.566786 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":210,"threads_started":1}
I20260925 01:26:22.977499 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:22.976762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:45759)
0925 01:26:22.977250 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:22.977336 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260925 01:26:23.390534 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:23.389805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:58391)
0925 01:26:23.390272 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:23.390357 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260925 01:26:23.568571 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:23.567862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:55523)
0925 01:26:23.568303 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:23.568385 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260925 01:26:23.979216 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:23.978420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:49103)
0925 01:26:23.978909 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:23.979016 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260925 01:26:24.392125 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:24.391400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:47117)
0925 01:26:24.391873 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:24.391955 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260925 01:26:24.570183 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:24.569426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:48097)
0925 01:26:24.569880 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:24.569969 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260925 01:26:24.980760 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:24.980020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:53095)
0925 01:26:24.980487 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:24.980568 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260925 01:26:25.393736 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:25.393024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:46595)
0925 01:26:25.393461 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:25.393541 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260925 01:26:25.571938 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:25.571225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:47193)
0925 01:26:25.571672 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:25.571758 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":215,"threads_started":1}
I20260925 01:26:25.982328 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:25.981581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:42325)
0925 01:26:25.982028 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:25.982118 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260925 01:26:26.395401 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:26.394658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:47955)
0925 01:26:26.395153 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:26.395234 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":270,"threads_started":1}
I20260925 01:26:26.573544 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:26.572854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:39187)
0925 01:26:26.573298 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:26.573380 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":204,"threads_started":1}
I20260925 01:26:26.983964 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:26.983179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:48411)
0925 01:26:26.983667 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:26.983782 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260925 01:26:27.397047 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:27.396309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:55209)
0925 01:26:27.396790 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:27.396886 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260925 01:26:27.575187 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:27.574433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:55953)
0925 01:26:27.574900 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:27.574978 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":228,"threads_started":1}
I20260925 01:26:27.985706 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:27.984867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:49275)
0925 01:26:27.985409 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:27.985493 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":219,"threads_started":1}
I20260925 01:26:28.398769 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:28.397950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:43749)
0925 01:26:28.398462 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:28.398546 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260925 01:26:28.576747 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:28.576004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:55795)
0925 01:26:28.576477 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:28.576587 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
I20260925 01:26:28.987385 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:28.986680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:41107)
0925 01:26:28.987137 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:28.987221 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260925 01:26:29.400337 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:29.399589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:34187)
0925 01:26:29.400057 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:29.400145 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":240,"threads_started":1}
I20260925 01:26:29.578428 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:29.577649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:42625)
0925 01:26:29.578125 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:29.578211 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260925 01:26:29.988890 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:29.988206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:54319)
0925 01:26:29.988631 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:29.988721 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":213,"threads_started":1}
I20260925 01:26:30.402096 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:30.401303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:49025)
0925 01:26:30.401841 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:30.401925 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
I20260925 01:26:30.580037 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:30.579330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:54807)
0925 01:26:30.579778 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:30.579861 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260925 01:26:30.990598 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:30.989809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:40519)
0925 01:26:30.990309 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:30.990422 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260925 01:26:31.403610 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:31.402926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:51033)
0925 01:26:31.403366 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:31.403452 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260925 01:26:31.581635 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:31.580910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:47323)
0925 01:26:31.581339 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:31.581432 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260925 01:26:31.992144 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:31.991386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:44931)
0925 01:26:31.991854 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:31.991939 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260925 01:26:32.405170 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:32.404472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:50481)
0925 01:26:32.404897 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:32.404991 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260925 01:26:32.583222 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:32.582521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:55059)
0925 01:26:32.582936 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:32.583066 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260925 01:26:32.993696 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:32.993010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:34129)
0925 01:26:32.993459 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:32.993542 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260925 01:26:33.406816 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:33.406107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:37257)
0925 01:26:33.406562 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:33.406646 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260925 01:26:33.584846 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:33.584093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:39585)
0925 01:26:33.584540 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:33.584648 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":213,"threads_started":1}
I20260925 01:26:33.995313 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:33.994569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:46871)
0925 01:26:33.995001 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:33.995112 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260925 01:26:34.408336 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:34.407621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:41129)
0925 01:26:34.408048 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:34.408141 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":212,"threads_started":1}
I20260925 01:26:34.586485 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:34.585744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:48415)
0925 01:26:34.586175 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:34.586302 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":208,"threads_started":1}
I20260925 01:26:34.996832 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:34.996125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:39015)
0925 01:26:34.996562 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:34.996644 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260925 01:26:35.409994 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:35.409258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:42539)
0925 01:26:35.409726 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:35.409814 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260925 01:26:35.588418 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:35.587573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.129:36873)
0925 01:26:35.588118 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:35.588240 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":242,"threads_started":1}
I20260925 01:26:35.998611 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:35.997799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.130:53509)
0925 01:26:35.998297 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:35.998392 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260925 01:26:36.396325 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22075
I20260925 01:26:36.411742 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:36.410982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:46041 (local address 127.6.236.131:54577)
0925 01:26:36.411431 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:36.411523 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":213,"threads_started":1}
I20260925 01:26:36.419245 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:36.397314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.6.236.129:33505)
0925 01:26:36.397853 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:36.397864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:36.398003 (+ 139us) server_negotiation.cc:407] Connection header received
0925 01:26:36.398326 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:36.398334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:36.398624 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:36.398814 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:36.400447 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.401753 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.402946 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.403612 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.403785 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:36.413299 (+ 9514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:36.413308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:36.413324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:36.413363 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:36.416144 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.416806 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.416814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.416822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.416999 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.417361 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.417365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.417367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.417898 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:36.418130 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:36.418551 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 01:26:36.418955 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":200,"threads_started":1}
I20260925 01:26:36.420557 22233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:36.502292 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22075
I20260925 01:26:36.517293 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22241
I20260925 01:26:36.534762 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:36.518182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.6.236.130:36703)
0925 01:26:36.518872 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:36.518890 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:36.518933 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:26:36.521137 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:36.521151 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:36.521674 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:36.521961 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:36.523316 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.524563 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.525794 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.526534 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.526674 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:36.529087 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:36.529095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:36.529110 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:36.529145 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:36.531797 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.532428 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.532434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.532440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.532601 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.532979 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.532983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.532985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.533472 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:36.533718 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:36.534077 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 01:26:36.534481 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":323,"threads_started":1}
I20260925 01:26:36.536043 22364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:36.617323 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22241
I20260925 01:26:36.631954 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22372
I20260925 01:26:36.648775 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:36.632880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.6.236.131:46263)
0925 01:26:36.633429 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:36.633442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:36.633482 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:26:36.635184 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:36.635194 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:36.635619 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:36.635916 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:36.637196 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.638454 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.639669 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:36.640326 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:36.640451 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:36.642866 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:36.642879 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:36.642901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:36.642954 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:36.645671 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.646461 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.646467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.646474 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.646633 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:36.646976 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:36.646980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:36.646982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:36.647538 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:36.647790 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:36.648068 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 01:26:36.648449 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
I20260925 01:26:36.650059 22495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:36.729666 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22372
2026-09-25T01:26:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23837 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:26:36 dist-test-slave-s0b2 krb5kdc[22557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:26:36 dist-test-slave-s0b2 krb5kdc[22557](info): set up 2 sockets
Sep 25 01:26:36 dist-test-slave-s0b2 krb5kdc[22557](info): commencing operation
krb5kdc: starting...
W20260925 01:26:37.747500 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:26:37 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:26:37.973136 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:38.286399 22570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:38.286671 22570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:38.286816 22570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:38.293926 22570 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:26:38.294020 22570 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:26:38.294059 22570 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:26:38.294130 22570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:26:38.294206 22570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:38.294317 22570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:38.294374 22570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:26:38.294446 22570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:26:38.304275 22570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:38.305510 22570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:38.306883 22570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:38.318064 22576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:38.318140 22575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:38.318110 22578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:38.318537 22570 server_base.cc:1034] running on GCE node
I20260925 01:26:38.319360 22570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:38.320675 22570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:38.321894 22570 hybrid_clock.cc:648] HybridClock initialized: now 1790299598321822 us; error 53 us; skew 500 ppm
I20260925 01:26:38.322192 22570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:38 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:38.325083 22570 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:26:38.326514 22570 webserver.cc:460] Webserver started at http://127.6.236.190:44903/ using document root <none> and password file <none>
I20260925 01:26:38.327155 22570 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:38.327256 22570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:38.327540 22570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:38.330416 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "c0c169ee232949d8aefa9bcec8da7603"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "4a3263a35c598cf2a3d439f77f79a1a3"
server_key_iv: "a367bab23b6732ac7682571e97bbda7d"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.331151 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "c0c169ee232949d8aefa9bcec8da7603"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "4a3263a35c598cf2a3d439f77f79a1a3"
server_key_iv: "a367bab23b6732ac7682571e97bbda7d"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.335974 22570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:26:38.339263 22585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:38.340515 22570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:26:38.340662 22570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "c0c169ee232949d8aefa9bcec8da7603"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "4a3263a35c598cf2a3d439f77f79a1a3"
server_key_iv: "a367bab23b6732ac7682571e97bbda7d"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.340857 22570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:38.360930 22570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:38.364770 22570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:38.365021 22570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:38.382331 22570 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:45867
I20260925 01:26:38.382344 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:45867 every 8 connection(s)
I20260925 01:26:38.383642 22570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:26:38.387290 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:38.393242 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22570
I20260925 01:26:38.393476 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:26:38.394330 7090 external_mini_cluster.cc:1428] Setting key 601849897673a6d889fe13dd55538b89
I20260925 01:26:38.398500 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap starting.
I20260925 01:26:38.401467 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:38.402568 22637 log.cc:824] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:38.405268 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: No bootstrap required, opened a new log
Sep 25 01:26:38 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:38.413161 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:26:38.413516 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:38.413609 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Initialized, Role: FOLLOWER
I20260925 01:26:38.414321 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:26:38.414565 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:26:38.414705 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:26:38.414870 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:38.416396 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:38.396529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.6.236.190:45867)
0925 01:26:38.397302 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:38.397314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:38.397355 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:26:38.399098 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:38.399119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:38.399495 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:38.399845 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:38.401289 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.402659 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.404301 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.404946 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.405119 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:38.409370 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:38.409378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:38.409395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:38.409440 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:38.412459 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:38.413139 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:38.413145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:38.413155 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:38.413315 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:38.413695 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:38.413700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:38.413703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:38.414202 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:38.414536 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:38.414954 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 01:26:38.415358 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":331,"threads_started":1}
I20260925 01:26:38.417567 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:26:38.418073 22637 leader_election.cc:302] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0c169ee232949d8aefa9bcec8da7603; no voters:
I20260925 01:26:38.418679 22637 leader_election.cc:288] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:26:38.418946 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:38.419817 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 LEADER]: Becoming Leader. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Running, Role: LEADER
I20260925 01:26:38.420390 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:26:38.420696 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:26:38.423440 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c169ee232949d8aefa9bcec8da7603. Latest consensus state: current_term: 1 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:26:38.423799 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:38.424322 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:26:38.424628 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:26:38.424799 22650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:26:38.433069 22650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:26:38.443810 22650 catalog_manager.cc:1347] Generated new cluster ID: b87ebebbe1b0416e8aa848e793dd83e9
I20260925 01:26:38.443920 22650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:26:38.464032 22650 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:26:38.465934 22650 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:26:38.481594 22650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Generated new TSK 0
I20260925 01:26:38.482752 22650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:26:38.550837 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:38.869112 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:38.869374 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:38.869506 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:38.876660 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:38.876765 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:38.876924 22664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:26:38.886479 22664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:38.887743 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:38.889029 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:38.901422 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:38.901362 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:38.901454 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:38.902189 22664 server_base.cc:1034] running on GCE node
I20260925 01:26:38.902853 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:38.903767 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:38.904982 22664 hybrid_clock.cc:648] HybridClock initialized: now 1790299598904916 us; error 53 us; skew 500 ppm
I20260925 01:26:38.905292 22664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:38 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:38.908278 22664 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:26:38.909675 22664 webserver.cc:460] Webserver started at http://127.6.236.129:37499/ using document root <none> and password file <none>
I20260925 01:26:38.910364 22664 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:38.910475 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:38.910770 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:38.913640 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "3be6294655b6424d9bccedcf2d453f1e"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "24d3b0b923a81f5e77b1cd70895277b3"
server_key_iv: "6cc0d7a2ffbce3d8f64509f470715765"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.914494 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "3be6294655b6424d9bccedcf2d453f1e"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "24d3b0b923a81f5e77b1cd70895277b3"
server_key_iv: "6cc0d7a2ffbce3d8f64509f470715765"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.919467 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:26:38.922868 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:38.924113 22664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:26:38.924252 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "3be6294655b6424d9bccedcf2d453f1e"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "24d3b0b923a81f5e77b1cd70895277b3"
server_key_iv: "6cc0d7a2ffbce3d8f64509f470715765"
server_key_version: "encryptionkey@0"
I20260925 01:26:38.924418 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:38.945694 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:38.949699 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:38.949973 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:38.950956 22664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:38.952211 22664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:38.952288 22664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:38.952389 22664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:38.952440 22664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:38.981839 22664 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:40693
I20260925 01:26:38.981932 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:40693 every 8 connection(s)
I20260925 01:26:38.983095 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:26:38.984256 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22664
I20260925 01:26:38.984459 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:26:38.985255 7090 external_mini_cluster.cc:1428] Setting key 0ef99a93098235745d9be75aa3785d99
Sep 25 01:26:38 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:39.003829 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:38.985748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:33209 (local address 127.6.236.190:45867)
0925 01:26:38.986233 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:38.986259 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:38.987257 (+ 998us) server_negotiation.cc:407] Connection header received
0925 01:26:38.989679 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:38.989683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:38.989750 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:38.989841 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:38.991687 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.992404 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.994583 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.994969 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.995071 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:38.998628 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:38.998634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:38.998637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:38.998670 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:39.000894 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.001589 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.001593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.001595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.001714 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.002118 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.002121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.002123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.002420 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:39.002556 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:39.003249 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 01:26:39.003475 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260925 01:26:39.004773 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:38.986341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:33209)
0925 01:26:38.987116 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:38.987166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:38.989385 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:38.990080 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:38.990090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:38.990794 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:38.991517 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:38.991540 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.992630 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.992636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:38.994455 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:38.994469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:38.995190 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:38.995197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:38.995460 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:38.996125 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:38.996157 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:38.998405 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:39.001022 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.001031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.001047 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.001455 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.001814 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.001819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.001823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.002001 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.002538 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:39.002546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:39.002877 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 01:26:39.003108 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 01:26:39.003631 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260925 01:26:39.006487 22792 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:26:39.006774 22792 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:39.007464 22792 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
I20260925 01:26:39.009761 22602 ts_manager.cc:194] Registered new tserver with Master: 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693)
I20260925 01:26:39.011260 22602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:33209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:26:39.035636 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:39.349913 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:39.350198 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:39.350404 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:39.357600 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:39.357726 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:39.357887 22798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:26:39.367614 22798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:39.368924 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:39.370270 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:39.383105 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:39.383215 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:39.383344 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:39.384212 22798 server_base.cc:1034] running on GCE node
I20260925 01:26:39.384824 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:39.386565 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:39.387780 22798 hybrid_clock.cc:648] HybridClock initialized: now 1790299599387726 us; error 57 us; skew 500 ppm
I20260925 01:26:39.388083 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:39 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:39.391052 22798 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:26:39.392590 22798 webserver.cc:460] Webserver started at http://127.6.236.130:40027/ using document root <none> and password file <none>
I20260925 01:26:39.393255 22798 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:39.393355 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:39.393639 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:39.396534 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "1a5baf361737425fbcafe5f3006249b1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "8746decc13602be7daa0aad8dfaec8cb"
server_key_iv: "36f1e6dd00028e250ceec4abb3922dac"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.397250 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "1a5baf361737425fbcafe5f3006249b1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "8746decc13602be7daa0aad8dfaec8cb"
server_key_iv: "36f1e6dd00028e250ceec4abb3922dac"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.402125 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:26:39.405390 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.406695 22798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:26:39.406842 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "1a5baf361737425fbcafe5f3006249b1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "8746decc13602be7daa0aad8dfaec8cb"
server_key_iv: "36f1e6dd00028e250ceec4abb3922dac"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.407016 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:39.439702 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:39.443941 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:39.444262 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:39.445192 22798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:39.446625 22798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:39.446697 22798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.446796 22798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:39.446837 22798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.475639 22798 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43659
I20260925 01:26:39.475685 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43659 every 8 connection(s)
I20260925 01:26:39.476837 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:26:39.486418 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22798
I20260925 01:26:39.486590 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:26:39.486995 7090 external_mini_cluster.cc:1428] Setting key ad6cf4e6394a01cdf08a80f2f584e2e1
Sep 25 01:26:39 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:39.495467 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:39.479299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:43053 (local address 127.6.236.190:45867)
0925 01:26:39.479476 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:39.479481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:39.480552 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 01:26:39.482423 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:39.482427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:39.482494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:39.482578 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:39.483931 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.484605 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.486004 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.486445 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.486552 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:39.490078 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:39.490083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:39.490086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:39.490116 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:39.492547 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.493349 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.493353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.493355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.493478 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.493888 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.493890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.493892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.494148 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:39.494295 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:39.495043 (+ 748us) server_negotiation.cc:299] Negotiation successful
0925 01:26:39.495234 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:26:39.496726 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:39.479747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:43053)
0925 01:26:39.480389 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:39.480431 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:39.482147 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:39.482764 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:39.482772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:39.483275 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:39.483801 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:39.483818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.484718 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.484723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:39.485901 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:39.485912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.486650 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.486658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:39.486868 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:39.487479 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:39.487504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:39.489884 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:39.492671 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.492678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.492695 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.493257 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.493566 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.493571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.493574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.493806 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.494277 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:39.494284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:39.494684 (+ 400us) client_negotiation.cc:769] Sending connection context
0925 01:26:39.494984 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 01:26:39.495465 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":249,"threads_started":1}
I20260925 01:26:39.498102 22926 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:26:39.498433 22926 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:39.499226 22926 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
I20260925 01:26:39.500697 22602 ts_manager.cc:194] Registered new tserver with Master: 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:26:39.501477 22602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:43053
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:26:39.539701 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:26:39.851821 22931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:26:39.852088 22931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:26:39.852229 22931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:26:39.859282 22931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:26:39.859385 22931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:26:39.859534 22931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:26:39.869048 22931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:26:39.870253 22931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:26:39.871490 22931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:26:39.883571 22936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:39.883606 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:26:39.883705 22939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:26:39.884528 22931 server_base.cc:1034] running on GCE node
I20260925 01:26:39.885094 22931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:26:39.885972 22931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:26:39.887162 22931 hybrid_clock.cc:648] HybridClock initialized: now 1790299599887120 us; error 29 us; skew 500 ppm
I20260925 01:26:39.887454 22931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:26:39 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:26:39.890372 22931 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:26:39.891654 22931 webserver.cc:460] Webserver started at http://127.6.236.131:35735/ using document root <none> and password file <none>
I20260925 01:26:39.892297 22931 fs_manager.cc:359] Metadata directory not provided
I20260925 01:26:39.892407 22931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:26:39.892692 22931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:26:39.895524 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "9a43366176124a9eaf7013b0183940f1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "b932239ba47432ef178d8b2719b99541"
server_key_iv: "a1bb7ae21fdae6dab11c2ad6e88510a4"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.896234 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "9a43366176124a9eaf7013b0183940f1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "b932239ba47432ef178d8b2719b99541"
server_key_iv: "a1bb7ae21fdae6dab11c2ad6e88510a4"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.901073 22931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:26:39.904318 22946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.905541 22931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 01:26:39.905684 22931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "9a43366176124a9eaf7013b0183940f1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "b932239ba47432ef178d8b2719b99541"
server_key_iv: "a1bb7ae21fdae6dab11c2ad6e88510a4"
server_key_version: "encryptionkey@0"
I20260925 01:26:39.905859 22931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:26:39.917975 22931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:26:39.921465 22931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:26:39.921723 22931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:26:39.922616 22931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:26:39.923820 22931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:26:39.923892 22931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.924001 22931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:26:39.924052 22931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:26:39.952390 22931 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:36043
I20260925 01:26:39.952425 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:36043 every 8 connection(s)
I20260925 01:26:39.953555 22931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:26:39.959224 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 22931
I20260925 01:26:39.959412 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:26:39.959836 7090 external_mini_cluster.cc:1428] Setting key 931809b18e5e18c53da7a10d3393bf6b
Sep 25 01:26:39 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:26:39.971802 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:39.955937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:47245 (local address 127.6.236.190:45867)
0925 01:26:39.956112 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:39.956117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:39.957185 (+ 1068us) server_negotiation.cc:407] Connection header received
0925 01:26:39.959021 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:39.959028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:39.959116 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:39.959227 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:39.960684 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.961763 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.963414 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.963816 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.963920 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:39.967060 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:39.967064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:39.967068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:39.967100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:39.969454 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.970046 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.970049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.970051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.970161 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:39.970510 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:39.970512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:39.970514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:39.970763 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:39.970867 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:39.971405 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 01:26:39.971556 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:26:39.972648 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:39.956378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:47245)
0925 01:26:39.957024 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:39.957065 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:39.958705 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:39.959430 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:39.959439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:39.959982 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:26:39.960536 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:39.960555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.961895 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.961901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:39.963285 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:39.963297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.964016 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.964023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:39.964250 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:39.964857 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:26:39.964878 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:26:39.966884 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:26:39.969558 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.969563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.969574 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.969961 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.970232 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:26:39.970259 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:26:39.970262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:26:39.970426 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:26:39.970847 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:26:39.970852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:26:39.971113 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 01:26:39.971312 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 01:26:39.971639 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":240,"threads_started":1}
I20260925 01:26:39.973884 23059 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:26:39.974162 23059 heartbeater.cc:464] Registering TS with master...
I20260925 01:26:39.974943 23059 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
I20260925 01:26:39.976142 22601 ts_manager.cc:194] Registered new tserver with Master: 9a43366176124a9eaf7013b0183940f1 (127.6.236.131:36043)
I20260925 01:26:39.976914 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:47245
I20260925 01:26:39.986662 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:26:40.002300 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:39.990868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.6.236.190:45867)
0925 01:26:39.991060 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:39.991065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:39.991471 (+ 406us) server_negotiation.cc:407] Connection header received
0925 01:26:39.991606 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:39.991610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:39.991667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:39.991737 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:39.992952 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.993675 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.995177 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:39.995578 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:39.995689 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:39.997851 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:39.997856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:39.997860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:39.997888 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:40.000187 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.000734 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.000737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.000740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.000860 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.001127 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.001130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.001131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.001416 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:40.001607 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.001833 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.001992 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:26:40.005966 22601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:26:40.012784 22601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:26:40.014009 22792 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
I20260925 01:26:40.032014 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.023746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40693 (local address 127.0.0.1:34512)
0925 01:26:40.024401 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:40.024440 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:40.024619 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:40.025648 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:40.025655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:40.025691 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:40.026096 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.026111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.027542 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.027566 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.029516 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.029531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.031038 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.031044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.031243 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.031298 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 01:26:40.031419 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 01:26:40.031635 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":464,"threads_started":1}
I20260925 01:26:40.033314 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.024021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.6.236.129:40693)
0925 01:26:40.024592 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.024601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.024631 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:26:40.024812 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.024821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.025075 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.025262 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:40.026310 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.027416 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.029658 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.030788 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.030913 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.031132 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.032842 (+ 1710us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.033001 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":946,"threads_started":1}
I20260925 01:26:40.037210 22727 tablet_service.cc:1467] Processing CreateTablet for tablet 37113af0614f4f4bbdac92f68d719340 (DEFAULT_TABLE table=test-table [id=94230005832146e0a5d37545dfa2297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:40.038026 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.026051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.6.236.130:43659)
0925 01:26:40.026572 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.026580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.031099 (+ 4519us) server_negotiation.cc:407] Connection header received
0925 01:26:40.031371 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.031378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.031580 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.031748 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:40.032509 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.033936 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.035694 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.036912 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.037035 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.037233 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.037551 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.037708 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":369,"threads_started":1}
I20260925 01:26:40.038115 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.024612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:36043 (local address 127.0.0.1:55060)
0925 01:26:40.025902 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:40.025915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:40.026004 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:40.031247 (+ 5243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:40.031252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:40.031277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:40.031681 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.031691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.033348 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.033353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.035016 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.035031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.037376 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.037382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.037539 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.037568 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:26:40.037635 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 01:26:40.037749 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":21,"thread_start_us":1010,"threads_started":1}
I20260925 01:26:40.038389 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.025538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.6.236.131:36043)
0925 01:26:40.030709 (+ 5171us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.030718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.030753 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:26:40.030854 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.030862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.031080 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.031275 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:40.031860 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.033233 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.035197 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.037018 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.037131 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.037335 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.037891 (+ 556us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.038044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5014,"thread_start_us":4867,"threads_started":1}
I20260925 01:26:40.038499 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.025028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43659 (local address 127.0.0.1:60078)
0925 01:26:40.030991 (+ 5963us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:40.031007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:40.031195 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:40.032003 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:40.032008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:40.032031 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:40.032333 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.032344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.034114 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.034119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.035548 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.035560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.037186 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.037192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.037360 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.037393 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:26:40.037486 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 01:26:40.037611 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5849,"mutex_wait_us":41,"thread_start_us":5727,"threads_started":1}
I20260925 01:26:40.039279 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37113af0614f4f4bbdac92f68d719340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:40.041926 22861 tablet_service.cc:1467] Processing CreateTablet for tablet 37113af0614f4f4bbdac92f68d719340 (DEFAULT_TABLE table=test-table [id=94230005832146e0a5d37545dfa2297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:40.041926 22994 tablet_service.cc:1467] Processing CreateTablet for tablet 37113af0614f4f4bbdac92f68d719340 (DEFAULT_TABLE table=test-table [id=94230005832146e0a5d37545dfa2297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:26:40.043295 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37113af0614f4f4bbdac92f68d719340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:40.043303 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37113af0614f4f4bbdac92f68d719340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:26:40.050441 23077 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Bootstrap starting.
I20260925 01:26:40.053439 23077 tablet_bootstrap.cc:654] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:40.054682 23077 log.cc:824] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:40.057190 23079 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Bootstrap starting.
I20260925 01:26:40.057387 23077 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: No bootstrap required, opened a new log
I20260925 01:26:40.057683 23077 ts_tablet_manager.cc:1397] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260925 01:26:40.058084 23080 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Bootstrap starting.
I20260925 01:26:40.060971 23079 tablet_bootstrap.cc:654] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:40.061363 23080 tablet_bootstrap.cc:654] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Neither blocks nor log segments found. Creating new log.
I20260925 01:26:40.061980 23079 log.cc:824] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:40.062626 23080 log.cc:824] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:26:40.066221 23080 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: No bootstrap required, opened a new log
I20260925 01:26:40.066404 23079 tablet_bootstrap.cc:492] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: No bootstrap required, opened a new log
I20260925 01:26:40.066505 23080 ts_tablet_manager.cc:1397] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260925 01:26:40.066660 23079 ts_tablet_manager.cc:1397] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:26:40.067642 23077 raft_consensus.cc:348] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.068094 23077 raft_consensus.cc:374] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:40.068230 23077 raft_consensus.cc:729] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be6294655b6424d9bccedcf2d453f1e, State: Initialized, Role: FOLLOWER
I20260925 01:26:40.068848 23077 consensus_queue.cc:260] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.070533 23077 ts_tablet_manager.cc:1428] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 01:26:40.074484 23080 raft_consensus.cc:348] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.074878 23080 raft_consensus.cc:374] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:40.074982 23080 raft_consensus.cc:729] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a43366176124a9eaf7013b0183940f1, State: Initialized, Role: FOLLOWER
I20260925 01:26:40.075589 23080 consensus_queue.cc:260] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.075784 23079 raft_consensus.cc:348] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.076167 23079 raft_consensus.cc:374] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:26:40.076267 23079 raft_consensus.cc:729] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5baf361737425fbcafe5f3006249b1, State: Initialized, Role: FOLLOWER
I20260925 01:26:40.076676 23059 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
I20260925 01:26:40.076915 23079 consensus_queue.cc:260] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.077122 23080 ts_tablet_manager.cc:1428] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260925 01:26:40.077862 22926 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
I20260925 01:26:40.078759 23079 ts_tablet_manager.cc:1428] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260925 01:26:40.205099 23060 tablet.cc:2369] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:40.208055 23084 raft_consensus.cc:482] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:26:40.208328 23084 raft_consensus.cc:504] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.210090 23084 leader_election.cc:288] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693), 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:26:40.216538 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.210570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36527 (local address 127.6.236.129:40693)
0925 01:26:40.210748 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.210753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.210771 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:26:40.210984 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.210991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.211093 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.211193 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:40.211784 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.212828 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.214679 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.215791 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.215906 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.215958 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.216270 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.216358 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:26:40.216639 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.210480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40693 (local address 127.6.236.131:36527)
0925 01:26:40.210676 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:40.210695 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:40.210827 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:40.211410 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:40.211414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:40.211448 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:40.211621 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.211632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.212954 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.212959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.214523 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.214535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.215924 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.215930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.216112 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.216147 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:26:40.216251 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:26:40.216384 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 01:26:40.217533 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.211120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:48081 (local address 127.6.236.130:43659)
0925 01:26:40.211319 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.211326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.211743 (+ 417us) server_negotiation.cc:407] Connection header received
0925 01:26:40.211887 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.211890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.211947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.212011 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:26:40.212603 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.213607 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.215534 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.216563 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.216729 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.216800 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.217247 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.217365 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 01:26:40.217576 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.210911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43659 (local address 127.6.236.131:48081)
0925 01:26:40.211642 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 01:26:40.211656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:26:40.211780 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:26:40.212044 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:26:40.212049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:26:40.212074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:26:40.212403 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.212415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.213749 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.213752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.215392 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:26:40.215407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.216942 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.216948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:26:40.217095 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.217118 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:26:40.217184 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 01:26:40.217286 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":416,"threads_started":1}
I20260925 01:26:40.217780 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37113af0614f4f4bbdac92f68d719340" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be6294655b6424d9bccedcf2d453f1e" is_pre_election: true
I20260925 01:26:40.218390 22747 raft_consensus.cc:2454] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 0.
I20260925 01:26:40.218570 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37113af0614f4f4bbdac92f68d719340" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5baf361737425fbcafe5f3006249b1" is_pre_election: true
I20260925 01:26:40.219079 22881 raft_consensus.cc:2454] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 0.
I20260925 01:26:40.219141 22947 leader_election.cc:302] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be6294655b6424d9bccedcf2d453f1e, 9a43366176124a9eaf7013b0183940f1; no voters:
I20260925 01:26:40.219552 23084 raft_consensus.cc:2781] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:26:40.219691 23084 raft_consensus.cc:482] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:26:40.219794 23084 raft_consensus.cc:3037] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:40.222034 23084 raft_consensus.cc:504] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:26:40.222594 23084 leader_election.cc:288] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693), 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:26:40.222995 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37113af0614f4f4bbdac92f68d719340" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be6294655b6424d9bccedcf2d453f1e"
I20260925 01:26:40.223034 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37113af0614f4f4bbdac92f68d719340" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5baf361737425fbcafe5f3006249b1"
I20260925 01:26:40.223212 22881 raft_consensus.cc:3037] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:40.223220 22747 raft_consensus.cc:3037] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:26:40.226341 22881 raft_consensus.cc:2454] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 1.
I20260925 01:26:40.226341 22747 raft_consensus.cc:2454] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 1.
I20260925 01:26:40.226727 22947 leader_election.cc:302] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be6294655b6424d9bccedcf2d453f1e, 9a43366176124a9eaf7013b0183940f1; no voters:
I20260925 01:26:40.226961 23084 raft_consensus.cc:2781] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:26:40.227378 23084 raft_consensus.cc:686] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a43366176124a9eaf7013b0183940f1, State: Running, Role: LEADER
I20260925 01:26:40.227880 23084 consensus_queue.cc:237] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
W20260925 01:26:40.228595 22927 tablet.cc:2369] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:40.232259 22602 catalog_manager.cc:5495] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a43366176124a9eaf7013b0183940f1 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "9a43366176124a9eaf7013b0183940f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } health_report { overall_health: HEALTHY } } }
W20260925 01:26:40.235212 22793 tablet.cc:2369] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:26:40.270190 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.265398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.6.236.131:36043)
0925 01:26:40.265610 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.265615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.265634 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:26:40.265827 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.265831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.265907 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.266021 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:26:40.266350 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.267029 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.268408 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.268793 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.268880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.269277 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:26:40.269746 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:26:40.269847 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.269887 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.269946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:26:40.277922 22881 raft_consensus.cc:1261] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Refusing update from remote peer 9a43366176124a9eaf7013b0183940f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:40.277926 22747 raft_consensus.cc:1261] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Refusing update from remote peer 9a43366176124a9eaf7013b0183940f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:26:40.279032 23087 consensus_queue.cc:1035] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:40.279399 23084 consensus_queue.cc:1035] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:26:40.290637 23090 mvcc.cc:205] Tried to move back new op lower bound from 7333067162728599552 to 7333067162537312256. Current Snapshot: MvccSnapshot[applied={T|T < 7333067162728599552}]
I20260925 01:26:40.303391 22602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:40.303627 22602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:26:40.306960 22602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Sending DeleteTablet for 3 replicas of tablet 37113af0614f4f4bbdac92f68d719340
I20260925 01:26:40.308030 22727 tablet_service.cc:1514] Processing DeleteTablet for tablet 37113af0614f4f4bbdac92f68d719340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:40 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:34512
I20260925 01:26:40.308418 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22664
I20260925 01:26:40.308501 22861 tablet_service.cc:1514] Processing DeleteTablet for tablet 37113af0614f4f4bbdac92f68d719340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:40 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:60078
I20260925 01:26:40.309079 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet 37113af0614f4f4bbdac92f68d719340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:26:40 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:55060
I20260925 01:26:40.319334 23103 tablet_replica.cc:330] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: stopping tablet replica
I20260925 01:26:40.319514 23100 tablet_replica.cc:330] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: stopping tablet replica
I20260925 01:26:40.319985 23100 raft_consensus.cc:2229] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:40.320492 23103 raft_consensus.cc:2229] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:26:40.320557 23100 raft_consensus.cc:2258] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:40.322398 23101 tablet_replica.cc:330] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: stopping tablet replica
I20260925 01:26:40.322367 23103 raft_consensus.cc:2258] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:40.322793 23101 raft_consensus.cc:2229] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:26:40.323283 23101 raft_consensus.cc:2258] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:26:40.324663 23100 ts_tablet_manager.cc:1905] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:40.326195 23101 ts_tablet_manager.cc:1905] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:40.326341 23103 ts_tablet_manager.cc:1905] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:26:40.331568 23100 ts_tablet_manager.cc:1918] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:40.331761 23100 log.cc:1196] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/37113af0614f4f4bbdac92f68d719340
I20260925 01:26:40.332310 23100 ts_tablet_manager.cc:1939] T 37113af0614f4f4bbdac92f68d719340 P 3be6294655b6424d9bccedcf2d453f1e: Deleting consensus metadata
Sep 25 01:26:40 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:26:40.333709 23103 ts_tablet_manager.cc:1918] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:40.334097 22586 catalog_manager.cc:4841] TS 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693): tablet 37113af0614f4f4bbdac92f68d719340 (table test-table [id=94230005832146e0a5d37545dfa2297c]) successfully deleted
I20260925 01:26:40.334491 23103 log.cc:1196] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/37113af0614f4f4bbdac92f68d719340
I20260925 01:26:40.335052 23103 ts_tablet_manager.cc:1939] T 37113af0614f4f4bbdac92f68d719340 P 9a43366176124a9eaf7013b0183940f1: Deleting consensus metadata
I20260925 01:26:40.336663 22587 catalog_manager.cc:4841] TS 9a43366176124a9eaf7013b0183940f1 (127.6.236.131:36043): tablet 37113af0614f4f4bbdac92f68d719340 (table test-table [id=94230005832146e0a5d37545dfa2297c]) successfully deleted
I20260925 01:26:40.338083 23101 ts_tablet_manager.cc:1918] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:26:40.338231 23101 log.cc:1196] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/37113af0614f4f4bbdac92f68d719340
I20260925 01:26:40.338672 23101 ts_tablet_manager.cc:1939] T 37113af0614f4f4bbdac92f68d719340 P 1a5baf361737425fbcafe5f3006249b1: Deleting consensus metadata
I20260925 01:26:40.340178 22586 catalog_manager.cc:4841] TS 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659): tablet 37113af0614f4f4bbdac92f68d719340 (table test-table [id=94230005832146e0a5d37545dfa2297c]) successfully deleted
I20260925 01:26:40.341828 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.309546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.6.236.129:40693)
0925 01:26:40.309716 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.309723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.321722 (+ 11999us) server_negotiation.cc:407] Connection header received
0925 01:26:40.321998 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.322004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.322082 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.322184 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:40.324154 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.325231 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.327171 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.327804 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.327916 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.335187 (+ 7271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:40.335201 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:40.335230 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:40.335272 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:40.338892 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.339825 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.339834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.339838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.339995 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.340478 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.340482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.340484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.340823 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:40.341035 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.341403 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.341571 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:26:40.342576 22787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:40.429919 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22664
I20260925 01:26:40.451972 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22798
Sep 25 01:26:40 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:26:40.468688 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.452785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.6.236.130:43659)
0925 01:26:40.452946 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.452951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.452968 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:26:40.453149 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.453153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.453233 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.453309 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:40.454976 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.455684 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.457593 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.457964 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.458053 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.463218 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:40.463232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:40.463248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:40.463282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:40.465927 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.466897 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.466904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.466906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.467023 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.467399 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.467402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.467404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.467753 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:40.467957 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.468266 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.468423 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:26:40.469301 22921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:40.557807 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22798
I20260925 01:26:40.572748 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22931
Sep 25 01:26:40 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:26:40.590991 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:26:40.573613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.6.236.131:36043)
0925 01:26:40.573812 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:26:40.573818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:26:40.574104 (+ 286us) server_negotiation.cc:407] Connection header received
0925 01:26:40.574356 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:26:40.574360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:26:40.574424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:26:40.574506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:26:40.576286 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.577094 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.579112 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:26:40.579565 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:26:40.579657 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:26:40.585114 (+ 5457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:26:40.585126 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:26:40.585141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:26:40.585174 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:26:40.588085 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.588937 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.588944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.588946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.589121 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:26:40.589530 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:26:40.589533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:26:40.589535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:26:40.589898 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:26:40.590121 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 01:26:40.590509 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 01:26:40.590679 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:26:40.591653 23054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:40.711602 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22931
I20260925 01:26:40.728080 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 22570
I20260925 01:26:40.729164 22632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:26:40.837604 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 22570
I20260925 01:27:00.856010 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--webserver_interface=127.6.236.190
--webserver_port=44903
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:01.175904 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:01.176203 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:01.176339 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:01.183462 23108 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:27:01.183557 23108 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:27:01.183621 23108 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:27:01.183694 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:01.183763 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:01.183853 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:01.183911 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:01.183983 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:01.195446 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=44903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:01.196692 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:01.197978 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:01.218477 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:01.218412 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:01.218458 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:01.219295 23108 server_base.cc:1034] running on GCE node
I20260925 01:27:01.220168 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:01.221663 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:01.222878 23108 hybrid_clock.cc:648] HybridClock initialized: now 1790299621222804 us; error 47 us; skew 500 ppm
I20260925 01:27:01.223251 23108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:01 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299621, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:01.226735 23108 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:27:01.228344 23108 webserver.cc:460] Webserver started at http://127.6.236.190:44903/ using document root <none> and password file <none>
I20260925 01:27:01.229072 23108 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:01.229199 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:01.235215 23108 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:27:01.238054 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:01.239384 23108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:27:01.239543 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "c0c169ee232949d8aefa9bcec8da7603"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "4a3263a35c598cf2a3d439f77f79a1a3"
server_key_iv: "a367bab23b6732ac7682571e97bbda7d"
server_key_version: "encryptionkey@0"
I20260925 01:27:01.240561 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:01.265915 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:01.270169 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:01.270462 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:01.287942 23108 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:45867
I20260925 01:27:01.287983 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:45867 every 8 connection(s)
I20260925 01:27:01.289551 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:01.296437 23176 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:27:01.297078 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23108
I20260925 01:27:01.298064 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:40693
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=37499
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:27:01.300973 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap starting.
I20260925 01:27:01.321853 23176 log.cc:824] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:01.333654 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:27:01.334131 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap complete.
I20260925 01:27:01.343031 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:01.343887 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Initialized, Role: FOLLOWER
I20260925 01:27:01.344463 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:01.344704 23176 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:01.344836 23176 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:01.344997 23176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:27:01.348822 23176 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:01.349342 23176 leader_election.cc:302] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0c169ee232949d8aefa9bcec8da7603; no voters:
I20260925 01:27:01.349937 23176 leader_election.cc:288] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:27:01.350075 23180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:27:01.351082 23180 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 LEADER]: Becoming Leader. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Running, Role: LEADER
I20260925 01:27:01.351680 23180 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:01.352205 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:01.353969 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c169ee232949d8aefa9bcec8da7603. Latest consensus state: current_term: 2 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:27:01.354365 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:01.354931 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:27:01.355244 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:01.357221 23186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:01.367408 23186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:01.368922 23186 catalog_manager.cc:1259] Loaded cluster ID: b87ebebbe1b0416e8aa848e793dd83e9
I20260925 01:27:01.369078 23186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:01.373262 23186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:01.374957 23186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Loaded TSK: 0
I20260925 01:27:01.422825 23186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Generated new TSK 1
I20260925 01:27:01.423266 23186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:27:01.648830 23178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:01.649113 23178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:01.649243 23178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:01.656497 23178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:01.656616 23178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:01.656764 23178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:01.666515 23178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:40693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=37499
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:01.667776 23178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:01.669059 23178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:01.683310 23207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:01.683336 23205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:01.683673 23204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:01.684387 23178 server_base.cc:1034] running on GCE node
I20260925 01:27:01.684964 23178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:01.686715 23178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:01.687940 23178 hybrid_clock.cc:648] HybridClock initialized: now 1790299621687885 us; error 46 us; skew 500 ppm
I20260925 01:27:01.688256 23178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:01 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299621, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:01.691431 23178 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:27:01.692961 23178 webserver.cc:460] Webserver started at http://127.6.236.129:37499/ using document root <none> and password file <none>
I20260925 01:27:01.693637 23178 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:01.693760 23178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:01.699123 23178 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:27:01.701821 23214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:01.703194 23178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:27:01.703351 23178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "3be6294655b6424d9bccedcf2d453f1e"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "24d3b0b923a81f5e77b1cd70895277b3"
server_key_iv: "6cc0d7a2ffbce3d8f64509f470715765"
server_key_version: "encryptionkey@0"
I20260925 01:27:01.704255 23178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:01.731184 23178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:01.735297 23178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:01.735546 23178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:01.736498 23178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:01.737898 23178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:01.737985 23178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:01.738095 23178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:01.738168 23178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:01.767879 23178 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:40693
I20260925 01:27:01.767947 23326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:40693 every 8 connection(s)
I20260925 01:27:01.769313 23178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:01.773850 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23178
I20260925 01:27:01.774506 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:43659
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=40027
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:27:01 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299621, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:01.793394 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:01.772315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:58151)
0925 01:27:01.773124 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:01.773170 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:01.774982 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:01.776608 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:01.776616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:01.777141 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:01.777667 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:01.777685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:01.779919 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:01.779923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:01.781182 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:01.781196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:01.782486 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:01.782493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:01.782767 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:01.783444 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:01.783469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:01.785857 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:01.789154 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:01.789163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:01.789178 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:01.789613 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:01.790021 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:01.790026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:01.790030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:01.790227 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:01.790968 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:01.790977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:01.791306 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 01:27:01.791538 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 01:27:01.792072 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":306,"threads_started":1}
I20260925 01:27:01.793394 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:01.772021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58151 (local address 127.6.236.190:45867)
0925 01:27:01.772546 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:01.772558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:01.773301 (+ 743us) server_negotiation.cc:407] Connection header received
0925 01:27:01.775584 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:01.775603 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:01.776055 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:01.776501 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:01.777864 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:01.779766 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:01.781326 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:01.782229 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:01.782392 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:01.786106 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:01.786114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:01.786128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:01.786169 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:01.789021 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:01.789734 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:01.789741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:01.789746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:01.789905 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:01.790345 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:01.790350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:01.790353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:01.790820 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:01.790987 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:01.791774 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 01:27:01.792156 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260925 01:27:01.796550 23327 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:27:01.796831 23327 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:01.797487 23327 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
I20260925 01:27:01.799770 23141 ts_manager.cc:194] Registered new tserver with Master: 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693)
I20260925 01:27:01.801782 23141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:58151
W20260925 01:27:02.088111 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:02.088399 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:02.088542 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:02.095670 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:02.095777 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:02.095934 23331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:02.105233 23331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:43659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=40027
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:02.106531 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:02.107808 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:02.119850 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:02.119782 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:02.119822 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:02.120824 23331 server_base.cc:1034] running on GCE node
I20260925 01:27:02.121364 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:02.122216 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:02.123435 23331 hybrid_clock.cc:648] HybridClock initialized: now 1790299622123391 us; error 37 us; skew 500 ppm
I20260925 01:27:02.123745 23331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:02 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:02.126641 23331 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:27:02.127913 23331 webserver.cc:460] Webserver started at http://127.6.236.130:40027/ using document root <none> and password file <none>
I20260925 01:27:02.128515 23331 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:02.128650 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:02.133526 23331 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:02.136068 23346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.137282 23331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 01:27:02.137420 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "1a5baf361737425fbcafe5f3006249b1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "8746decc13602be7daa0aad8dfaec8cb"
server_key_iv: "36f1e6dd00028e250ceec4abb3922dac"
server_key_version: "encryptionkey@0"
I20260925 01:27:02.138325 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:02.157961 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:02.161994 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:02.162252 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:02.163165 23331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:02.164485 23331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:02.164577 23331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.164690 23331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:02.164758 23331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.199267 23331 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43659
I20260925 01:27:02.199348 23458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43659 every 8 connection(s)
I20260925 01:27:02.201197 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:02.207937 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23331
I20260925 01:27:02.208707 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:36043
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=35735
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:27:02 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:02.222873 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.205521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:50875 (local address 127.6.236.190:45867)
0925 01:27:02.205722 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.205728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.205752 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:27:02.207796 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.207802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.207898 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.208006 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:02.209944 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.210846 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.212829 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.213394 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.213470 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.217094 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:02.217099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:02.217103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:02.217141 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:02.219997 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.220661 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.220665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.220669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.220804 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.221166 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.221170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.221173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.221491 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:02.221654 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.222309 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.222540 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:27:02.223713 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.204768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:50875)
0925 01:27:02.205533 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.205594 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.207477 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.208279 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.208291 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.208989 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:02.209712 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.209736 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.211017 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.211023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.212658 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.212673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.213750 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.213755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.213943 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.214805 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:02.214831 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:02.216872 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:02.220124 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:02.220131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:02.220143 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:02.220552 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:02.220897 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:02.220900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:02.220902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:02.221080 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:02.221609 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:02.221614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:02.221945 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.222172 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.222603 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":920,"threads_started":1}
I20260925 01:27:02.225103 23459 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:27:02.225375 23459 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:02.226040 23459 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
I20260925 01:27:02.227463 23141 ts_manager.cc:194] Registered new tserver with Master: 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:27:02.228214 23141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:50875
W20260925 01:27:02.527199 23462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:02.527456 23462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:02.527590 23462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:02.534526 23462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:02.534633 23462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:02.534785 23462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:02.543998 23462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:36043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=35735
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:02.545153 23462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:02.546418 23462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:02.558447 23468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:02.558491 23470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:02.558517 23467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:02.559373 23462 server_base.cc:1034] running on GCE node
I20260925 01:27:02.559883 23462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:02.560703 23462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:02.561889 23462 hybrid_clock.cc:648] HybridClock initialized: now 1790299622561844 us; error 34 us; skew 500 ppm
I20260925 01:27:02.562186 23462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:02 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:02.565006 23462 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:27:02.566365 23462 webserver.cc:460] Webserver started at http://127.6.236.131:35735/ using document root <none> and password file <none>
I20260925 01:27:02.566968 23462 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:02.567075 23462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:02.572154 23462 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:27:02.574651 23477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.575863 23462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:02.575999 23462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "9a43366176124a9eaf7013b0183940f1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "b932239ba47432ef178d8b2719b99541"
server_key_iv: "a1bb7ae21fdae6dab11c2ad6e88510a4"
server_key_version: "encryptionkey@0"
I20260925 01:27:02.576911 23462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:02.598712 23462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:02.602499 23462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:02.602794 23462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:02.603955 23462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:02.605482 23462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:02.605577 23462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.605679 23462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:02.605741 23462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:02.634972 23462 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:36043
I20260925 01:27:02.635038 23589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:36043 every 8 connection(s)
I20260925 01:27:02.636333 23462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:02.642522 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23462
Sep 25 01:27:02 dist-test-slave-s0b2 krb5kdc[22557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:02.657887 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.638754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36097 (local address 127.6.236.190:45867)
0925 01:27:02.638964 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.638970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.640087 (+ 1117us) server_negotiation.cc:407] Connection header received
0925 01:27:02.641886 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.641890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.641976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.642070 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:02.644080 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.645119 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.646937 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.647522 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.647655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.651587 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:02.651592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:02.651596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:02.651626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:02.654599 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.655439 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.655443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.655445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.655557 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.655992 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.655997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.656000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.656341 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:02.656450 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.657303 (+ 853us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.657520 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:27:02.658320 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.643466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.6.236.190:45867)
0925 01:27:02.643978 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.643984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.644275 (+ 291us) server_negotiation.cc:407] Connection header received
0925 01:27:02.644401 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.644406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.644480 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.644577 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:02.646090 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.647127 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.648748 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.649367 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.649489 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.652518 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:02.652523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:02.652526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:02.652570 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:02.655586 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.656252 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.656256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.656259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.656383 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.656748 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.656753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.656756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.657065 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:02.657331 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.657656 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.657843 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"spinlock_wait_cycles":10112,"thread_start_us":280,"threads_started":1}
I20260925 01:27:02.658931 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.639254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:36097)
0925 01:27:02.639936 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.639973 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.641642 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.642382 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.642393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.643130 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:02.643863 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.643890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.645265 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.645269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.646789 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.646803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.647718 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.647722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.647918 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.648539 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:02.648560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:02.651379 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:02.654729 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:02.654739 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:02.654754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:02.655323 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:02.655660 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:02.655665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:02.655669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:02.655864 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:02.656462 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:02.656470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:02.656893 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.657181 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.657664 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":259,"threads_started":1}
I20260925 01:27:02.660130 23590 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45867
I20260925 01:27:02.660423 23590 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:02.661161 23590 heartbeater.cc:511] Master 127.6.236.190:45867 requested a full tablet report, sending...
I20260925 01:27:02.662519 23140 ts_manager.cc:194] Registered new tserver with Master: 9a43366176124a9eaf7013b0183940f1 (127.6.236.131:36043)
I20260925 01:27:02.663336 23140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:36097
I20260925 01:27:02.672730 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:02.679028 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.673575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.6.236.190:45867)
0925 01:27:02.673728 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.673733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.674190 (+ 457us) server_negotiation.cc:407] Connection header received
0925 01:27:02.674449 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.674452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.674570 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.674685 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:02.675136 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.675799 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.677224 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.677620 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.677723 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.678221 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:02.678461 (+ 240us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 01:27:02.678780 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33806: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 01:27:02.679137 23593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33806: token expired
W20260925 01:27:02.679656 23064 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.236.190:45867)
I20260925 01:27:02.691488 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.680077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.6.236.190:45867)
0925 01:27:02.680197 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.680201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.680231 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:27:02.680359 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.680363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.680413 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.680483 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:02.681819 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.682608 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.684003 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.684368 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.684448 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.686910 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:02.686914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:02.686917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:02.686947 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:02.689324 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.689964 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.689968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.689971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.690090 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.690419 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.690422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.690424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.690714 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:02.690897 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.691123 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.691279 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 01:27:02.694694 23140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:02.701658 23140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:02.723785 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.712960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.6.236.129:40693)
0925 01:27:02.714478 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.714489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.714519 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:27:02.714621 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.714629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.714885 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.715089 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:02.716931 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.718469 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.721243 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.722714 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.722844 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.723058 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.723332 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.723500 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":1427,"threads_started":1}
I20260925 01:27:02.723887 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.712885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40693 (local address 127.0.0.1:36188)
0925 01:27:02.713672 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.713711 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.713885 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.715679 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.715690 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.715732 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:02.716249 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.716263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.718648 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.718654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.721054 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.721082 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.722923 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.722929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.723122 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.723160 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.723271 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.723474 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":29,"thread_start_us":359,"threads_started":1}
I20260925 01:27:02.724321 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.713520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:36043 (local address 127.0.0.1:47910)
0925 01:27:02.714929 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.714945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.715058 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.717019 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.717025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.717054 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:02.717454 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.717466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.719610 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.719615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.721805 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.721818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.723529 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.723535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.723715 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.723754 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.723823 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.723951 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"mutex_wait_us":76,"thread_start_us":3603,"threads_started":1}
I20260925 01:27:02.725109 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.714516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.6.236.131:36043)
0925 01:27:02.716382 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.716391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.716428 (+ 37us) server_negotiation.cc:407] Connection header received
0925 01:27:02.716527 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.716535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.716796 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.717641 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:02.717934 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.719466 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.721989 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.723294 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.724369 (+ 1075us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.724583 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.724642 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.724786 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":1545,"threads_started":1}
I20260925 01:27:02.725137 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.714097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43659 (local address 127.0.0.1:56324)
0925 01:27:02.716641 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.716659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.716778 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.718648 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.718653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.718679 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:02.719138 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.719149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.720471 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.720477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.722441 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.722458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.724497 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.724503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.724670 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.724702 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.724768 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.724867 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"mutex_wait_us":2,"thread_start_us":1427,"threads_started":1}
I20260925 01:27:02.726033 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.715290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.6.236.130:43659)
0925 01:27:02.717786 (+ 2496us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.717794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.717827 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:27:02.717924 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.717932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.718136 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.718354 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:02.719310 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.720335 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.722627 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.724006 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.725323 (+ 1317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.725536 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.725601 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.725745 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2320,"thread_start_us":2182,"threads_started":1}
I20260925 01:27:02.728215 23262 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f (DEFAULT_TABLE table=test-table [id=00116139d9124898a9d364c484045eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:02.728938 23394 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f (DEFAULT_TABLE table=test-table [id=00116139d9124898a9d364c484045eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:02.728984 23525 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f (DEFAULT_TABLE table=test-table [id=00116139d9124898a9d364c484045eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:02.729709 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7cc748a7484a3e8b5f7825126adc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:02.730049 23394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7cc748a7484a3e8b5f7825126adc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:02.730336 23525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7cc748a7484a3e8b5f7825126adc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:02.741355 23606 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Bootstrap starting.
I20260925 01:27:02.745115 23606 tablet_bootstrap.cc:654] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:02.746093 23606 log.cc:824] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:02.748629 23606 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: No bootstrap required, opened a new log
I20260925 01:27:02.748852 23606 ts_tablet_manager.cc:1397] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 01:27:02.750931 23607 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Bootstrap starting.
I20260925 01:27:02.751184 23608 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Bootstrap starting.
I20260925 01:27:02.754541 23607 tablet_bootstrap.cc:654] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:02.754729 23608 tablet_bootstrap.cc:654] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:02.755533 23607 log.cc:824] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:02.755852 23608 log.cc:824] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:02.757583 23606 raft_consensus.cc:348] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.758010 23606 raft_consensus.cc:374] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:02.758121 23606 raft_consensus.cc:729] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5baf361737425fbcafe5f3006249b1, State: Initialized, Role: FOLLOWER
I20260925 01:27:02.758374 23608 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: No bootstrap required, opened a new log
I20260925 01:27:02.758466 23607 tablet_bootstrap.cc:492] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: No bootstrap required, opened a new log
I20260925 01:27:02.758668 23608 ts_tablet_manager.cc:1397] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 01:27:02.758703 23607 ts_tablet_manager.cc:1397] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 01:27:02.758738 23606 consensus_queue.cc:260] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.759835 23459 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
I20260925 01:27:02.760411 23606 ts_tablet_manager.cc:1428] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 01:27:02.767010 23608 raft_consensus.cc:348] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.767460 23608 raft_consensus.cc:374] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:02.767238 23607 raft_consensus.cc:348] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.767578 23608 raft_consensus.cc:729] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be6294655b6424d9bccedcf2d453f1e, State: Initialized, Role: FOLLOWER
I20260925 01:27:02.767670 23607 raft_consensus.cc:374] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:02.767771 23607 raft_consensus.cc:729] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a43366176124a9eaf7013b0183940f1, State: Initialized, Role: FOLLOWER
I20260925 01:27:02.768185 23608 consensus_queue.cc:260] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.768334 23607 consensus_queue.cc:260] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.769719 23590 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
I20260925 01:27:02.769886 23608 ts_tablet_manager.cc:1428] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 01:27:02.770354 23607 ts_tablet_manager.cc:1428] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260925 01:27:02.770448 23327 heartbeater.cc:503] Master 127.6.236.190:45867 was elected leader, sending a full tablet report...
W20260925 01:27:02.771328 23328 tablet.cc:2369] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:02.793759 23614 raft_consensus.cc:482] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:02.793982 23614 raft_consensus.cc:504] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.795482 23614 leader_election.cc:288] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693), 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:27:02.801781 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.795782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:40693 (local address 127.6.236.131:42641)
0925 01:27:02.795987 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.796006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.796127 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.796557 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.796562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.796591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:02.796787 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.796801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.797987 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.797991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.799906 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.799922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.801209 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.801213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.801350 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.801373 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.801440 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.801529 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":24}
I20260925 01:27:02.801870 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.795783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:42641 (local address 127.6.236.129:40693)
0925 01:27:02.795961 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.795968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.796096 (+ 128us) server_negotiation.cc:407] Connection header received
0925 01:27:02.796267 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.796273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.796347 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.796442 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:02.796934 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.797868 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.800054 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.801065 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.801205 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.801277 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.801487 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.801619 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:27:02.802520 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.796264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:36351 (local address 127.6.236.130:43659)
0925 01:27:02.796457 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.796462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.796916 (+ 454us) server_negotiation.cc:407] Connection header received
0925 01:27:02.797050 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.797055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.797130 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.797220 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:02.797626 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.798406 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.800460 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.801579 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.801732 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.801801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.802162 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.802297 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:27:02.802536 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.796119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:43659 (local address 127.6.236.131:36351)
0925 01:27:02.796816 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:02.796832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:02.796940 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:02.797223 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:02.797225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:02.797241 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:02.797501 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.797508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.798527 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.798532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.800321 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:02.800331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.801780 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.801786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:02.801956 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.801989 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:27:02.802116 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 01:27:02.802228 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":1,"thread_start_us":367,"threads_started":1}
I20260925 01:27:02.802881 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7cc748a7484a3e8b5f7825126adc3f" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be6294655b6424d9bccedcf2d453f1e" is_pre_election: true
I20260925 01:27:02.803395 23282 raft_consensus.cc:2454] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 0.
I20260925 01:27:02.803457 23414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7cc748a7484a3e8b5f7825126adc3f" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5baf361737425fbcafe5f3006249b1" is_pre_election: true
I20260925 01:27:02.803884 23414 raft_consensus.cc:2454] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 0.
I20260925 01:27:02.803973 23478 leader_election.cc:302] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be6294655b6424d9bccedcf2d453f1e, 9a43366176124a9eaf7013b0183940f1; no voters:
I20260925 01:27:02.804334 23614 raft_consensus.cc:2781] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:02.804462 23614 raft_consensus.cc:482] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:02.804557 23614 raft_consensus.cc:3037] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:02.806789 23614 raft_consensus.cc:504] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.807315 23614 leader_election.cc:288] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693), 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659)
I20260925 01:27:02.807719 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7cc748a7484a3e8b5f7825126adc3f" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be6294655b6424d9bccedcf2d453f1e"
I20260925 01:27:02.807833 23414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7cc748a7484a3e8b5f7825126adc3f" candidate_uuid: "9a43366176124a9eaf7013b0183940f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a5baf361737425fbcafe5f3006249b1"
I20260925 01:27:02.807925 23282 raft_consensus.cc:3037] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:02.808050 23414 raft_consensus.cc:3037] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:02.811069 23414 raft_consensus.cc:2454] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 1.
I20260925 01:27:02.811069 23282 raft_consensus.cc:2454] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a43366176124a9eaf7013b0183940f1 in term 1.
I20260925 01:27:02.811419 23478 leader_election.cc:302] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be6294655b6424d9bccedcf2d453f1e, 9a43366176124a9eaf7013b0183940f1; no voters:
I20260925 01:27:02.811662 23614 raft_consensus.cc:2781] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:02.812050 23614 raft_consensus.cc:686] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 1 LEADER]: Becoming Leader. State: Replica: 9a43366176124a9eaf7013b0183940f1, State: Running, Role: LEADER
I20260925 01:27:02.812484 23614 consensus_queue.cc:237] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } }
I20260925 01:27:02.816696 23141 catalog_manager.cc:5495] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a43366176124a9eaf7013b0183940f1 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "9a43366176124a9eaf7013b0183940f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a43366176124a9eaf7013b0183940f1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 36043 } health_report { overall_health: HEALTHY } } }
I20260925 01:27:02.845340 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.840301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.6.236.131:36043)
0925 01:27:02.840456 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.840460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.840541 (+ 81us) server_negotiation.cc:407] Connection header received
0925 01:27:02.840712 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.840716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.840803 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.840921 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:02.841252 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.842019 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.843383 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.843818 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.843909 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.844331 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:02.844843 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:02.844946 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.845003 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.845064 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:27:02.853248 23414 raft_consensus.cc:1261] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Refusing update from remote peer 9a43366176124a9eaf7013b0183940f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:02.853259 23282 raft_consensus.cc:1261] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Refusing update from remote peer 9a43366176124a9eaf7013b0183940f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:02.854183 23616 consensus_queue.cc:1035] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a5baf361737425fbcafe5f3006249b1" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:02.854547 23614 consensus_queue.cc:1035] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be6294655b6424d9bccedcf2d453f1e" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:02.865178 23618 mvcc.cc:205] Tried to move back new op lower bound from 7333067255196499968 to 7333067255043465216. Current Snapshot: MvccSnapshot[applied={T|T < 7333067255196499968}]
I20260925 01:27:02.879820 23140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:02.880060 23140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:02.883438 23140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Sending DeleteTablet for 3 replicas of tablet 9b7cc748a7484a3e8b5f7825126adc3f
I20260925 01:27:02.884599 23262 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:02 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:36188
I20260925 01:27:02.884892 23394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:02 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:56324
I20260925 01:27:02.885250 23525 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7cc748a7484a3e8b5f7825126adc3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:02 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:47910
I20260925 01:27:02.885499 23628 tablet_replica.cc:330] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: stopping tablet replica
I20260925 01:27:02.885944 23628 raft_consensus.cc:2229] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:02.886107 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23178
I20260925 01:27:02.886580 23628 raft_consensus.cc:2258] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:02.897274 23628 ts_tablet_manager.cc:1905] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:02.898736 23630 tablet_replica.cc:330] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: stopping tablet replica
I20260925 01:27:02.898970 23629 tablet_replica.cc:330] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: stopping tablet replica
I20260925 01:27:02.899121 23630 raft_consensus.cc:2229] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:02.899367 23629 raft_consensus.cc:2229] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:02.899677 23630 raft_consensus.cc:2258] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:02.899842 23629 raft_consensus.cc:2258] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:02.901551 23630 ts_tablet_manager.cc:1905] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:02.901794 23629 ts_tablet_manager.cc:1905] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:02.904865 23628 ts_tablet_manager.cc:1918] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:02.905117 23628 log.cc:1196] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/9b7cc748a7484a3e8b5f7825126adc3f
I20260925 01:27:02.906044 23628 ts_tablet_manager.cc:1939] T 9b7cc748a7484a3e8b5f7825126adc3f P 3be6294655b6424d9bccedcf2d453f1e: Deleting consensus metadata
I20260925 01:27:02.907951 23125 catalog_manager.cc:4841] TS 3be6294655b6424d9bccedcf2d453f1e (127.6.236.129:40693): tablet 9b7cc748a7484a3e8b5f7825126adc3f (table test-table [id=00116139d9124898a9d364c484045eaa]) successfully deleted
I20260925 01:27:02.908381 23629 ts_tablet_manager.cc:1918] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:02.908530 23629 log.cc:1196] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/9b7cc748a7484a3e8b5f7825126adc3f
I20260925 01:27:02.909060 23629 ts_tablet_manager.cc:1939] T 9b7cc748a7484a3e8b5f7825126adc3f P 1a5baf361737425fbcafe5f3006249b1: Deleting consensus metadata
I20260925 01:27:02.910545 23125 catalog_manager.cc:4841] TS 1a5baf361737425fbcafe5f3006249b1 (127.6.236.130:43659): tablet 9b7cc748a7484a3e8b5f7825126adc3f (table test-table [id=00116139d9124898a9d364c484045eaa]) successfully deleted
I20260925 01:27:02.910868 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:02.887351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.6.236.129:40693)
0925 01:27:02.887555 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:02.887562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:02.887586 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:27:02.887646 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:02.887652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:02.887741 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:02.887854 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:02.890498 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.891636 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.893791 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:02.894471 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:02.894591 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:02.903548 (+ 8957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:02.903563 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:02.903581 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:02.903623 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:02.907568 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.908479 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.908501 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.908505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.908657 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:02.909122 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:02.909127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:02.909131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:02.909601 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:02.909899 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:02.910292 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 01:27:02.910522 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:27:02.911665 23322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:02.914116 23630 ts_tablet_manager.cc:1918] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:02.914302 23630 log.cc:1196] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/9b7cc748a7484a3e8b5f7825126adc3f
I20260925 01:27:02.914819 23630 ts_tablet_manager.cc:1939] T 9b7cc748a7484a3e8b5f7825126adc3f P 9a43366176124a9eaf7013b0183940f1: Deleting consensus metadata
I20260925 01:27:02.916352 23126 catalog_manager.cc:4841] TS 9a43366176124a9eaf7013b0183940f1 (127.6.236.131:36043): tablet 9b7cc748a7484a3e8b5f7825126adc3f (table test-table [id=00116139d9124898a9d364c484045eaa]) successfully deleted
I20260925 01:27:03.025110 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23178
I20260925 01:27:03.047268 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23331
I20260925 01:27:03.060571 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:03.048059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.6.236.130:43659)
0925 01:27:03.048227 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:03.048232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:03.048252 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:27:03.048434 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:03.048439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:03.048507 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:03.048591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:03.049913 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:03.050720 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:03.052042 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:03.052530 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:03.052644 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:03.055283 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:03.055297 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:03.055318 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:03.055361 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:03.057923 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:03.058626 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:03.058634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:03.058636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:03.058764 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:03.059108 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:03.059112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:03.059114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:03.059473 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:03.059736 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:03.060042 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 01:27:03.060244 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:27:03.061205 23454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:03.153131 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23331
I20260925 01:27:03.167914 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23462
I20260925 01:27:03.181885 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:03.168661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.6.236.131:36043)
0925 01:27:03.168852 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:03.168858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:03.168947 (+ 89us) server_negotiation.cc:407] Connection header received
0925 01:27:03.169137 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:03.169146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:03.169213 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:03.169315 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:03.170746 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:03.171554 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:03.172815 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:03.173270 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:03.173400 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:03.175983 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:03.175996 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:03.176012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:03.176048 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:03.179041 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:03.179713 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:03.179720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:03.179723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:03.179896 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:03.180278 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:03.180282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:03.180284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:03.180700 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:03.180987 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:03.181287 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 01:27:03.181504 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:27:03.182552 23585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:03.310714 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23462
I20260925 01:27:03.327316 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23108
I20260925 01:27:03.328352 23171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:03.438822 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23108
I20260925 01:27:23.457679 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--webserver_interface=127.6.236.190
--webserver_port=44903
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:23.779506 23635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:23.779801 23635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:23.779932 23635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:23.787046 23635 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:27:23.787139 23635 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:27:23.787209 23635 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:27:23.787276 23635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:23.787348 23635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:23.787439 23635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:23.787495 23635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:23.787566 23635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:23.797647 23635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=44903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:23.799018 23635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:23.800313 23635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:23.811705 23640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:23.811800 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:23.811793 23643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:23.812784 23635 server_base.cc:1034] running on GCE node
I20260925 01:27:23.813715 23635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:23.815449 23635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:23.816649 23635 hybrid_clock.cc:648] HybridClock initialized: now 1790299643816582 us; error 35 us; skew 500 ppm
I20260925 01:27:23.817046 23635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:23 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:23.826133 23635 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:27:23.827531 23635 webserver.cc:460] Webserver started at http://127.6.236.190:44903/ using document root <none> and password file <none>
I20260925 01:27:23.828208 23635 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:23.828315 23635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:23.833853 23635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:23.836674 23650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:23.837949 23635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:27:23.838109 23635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "c0c169ee232949d8aefa9bcec8da7603"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "4a3263a35c598cf2a3d439f77f79a1a3"
server_key_iv: "a367bab23b6732ac7682571e97bbda7d"
server_key_version: "encryptionkey@0"
I20260925 01:27:23.839147 23635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:23.857228 23635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:23.861455 23635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:23.861747 23635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:23.878856 23635 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:45867
I20260925 01:27:23.878875 23701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:45867 every 8 connection(s)
I20260925 01:27:23.880405 23635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:23.887161 23702 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:27:23.889065 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23635
I20260925 01:27:23.889927 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:40693
--local_ip_for_outbound_sockets=127.6.236.129
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=37499
--webserver_interface=127.6.236.129
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:27:23.890488 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap starting.
I20260925 01:27:23.915071 23702 log.cc:824] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:23.935563 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:27:23.936183 23702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Bootstrap complete.
I20260925 01:27:23.949399 23702 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:23.950644 23702 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Initialized, Role: FOLLOWER
I20260925 01:27:23.951352 23702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:23.951628 23702 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:23.951746 23702 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:23.951922 23702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:27:23.957737 23702 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:23.958364 23702 leader_election.cc:302] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0c169ee232949d8aefa9bcec8da7603; no voters:
I20260925 01:27:23.959025 23702 leader_election.cc:288] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:27:23.959149 23706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:27:23.960336 23706 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [term 3 LEADER]: Becoming Leader. State: Replica: c0c169ee232949d8aefa9bcec8da7603, State: Running, Role: LEADER
I20260925 01:27:23.960903 23706 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } }
I20260925 01:27:23.961100 23702 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:23.963171 23708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c169ee232949d8aefa9bcec8da7603. Latest consensus state: current_term: 3 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:27:23.963692 23707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c0c169ee232949d8aefa9bcec8da7603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c169ee232949d8aefa9bcec8da7603" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45867 } } }
I20260925 01:27:23.964052 23708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:23.964426 23707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:23.973978 23712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:23.982308 23712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:23.983753 23712 catalog_manager.cc:1259] Loaded cluster ID: b87ebebbe1b0416e8aa848e793dd83e9
I20260925 01:27:23.983906 23712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:23.987912 23712 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:23.989471 23712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Loaded TSK: 0
I20260925 01:27:23.989677 23712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0c169ee232949d8aefa9bcec8da7603: Loaded TSK: 1
F20260925 01:27:23.990963 23712 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4f1ae969ad google::LogMessage::Fail() at ??:0
@ 0x7f4f1ae9988a google::LogMessage::SendToLog() at ??:0
@ 0x7f4f1ae964de google::LogMessage::Flush() at ??:0
@ 0x7f4f1ae97a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4f1ebfa7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4f1ebe973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4f308c51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4f308c4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4f309502a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f4f309500c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f4f3099a5e9 std::function<>::operator()() at ??:0
@ 0x7f4f308c8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4f308c6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4f3094e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4f3094e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4f1dae2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4f1dae591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4f1dae5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4f1dac0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f20d9a6db start_thread at ??:0
@ 0x7f4f19d3061f clone at ??:0
W20260925 01:27:24.286491 23704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:24.286777 23704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:24.286911 23704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:24.294296 23704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:24.294420 23704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:24.294592 23704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:24.304463 23704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:40693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=37499
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:24.305734 23704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:24.307112 23704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:24.320068 23749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:24.320039 23752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:24.320108 23750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:24.320779 23704 server_base.cc:1034] running on GCE node
I20260925 01:27:24.321352 23704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:24.322301 23704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:24.323493 23704 hybrid_clock.cc:648] HybridClock initialized: now 1790299644323438 us; error 49 us; skew 500 ppm
I20260925 01:27:24.323805 23704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:24 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:24.326812 23704 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:27:24.328203 23704 webserver.cc:460] Webserver started at http://127.6.236.129:37499/ using document root <none> and password file <none>
I20260925 01:27:24.328889 23704 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:24.329000 23704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:24.334314 23704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:27:24.336972 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.338320 23704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:27:24.338474 23704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "3be6294655b6424d9bccedcf2d453f1e"
format_stamp: "Formatted at 2026-09-25 01:26:38 on dist-test-slave-s0b2"
server_key: "24d3b0b923a81f5e77b1cd70895277b3"
server_key_iv: "6cc0d7a2ffbce3d8f64509f470715765"
server_key_version: "encryptionkey@0"
I20260925 01:27:24.339438 23704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:24.353636 23704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:24.357681 23704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:24.357959 23704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:24.359189 23704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:24.360589 23704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:24.360684 23704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.360805 23704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:24.360888 23704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.392166 23704 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:40693
I20260925 01:27:24.392230 23871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:40693 every 8 connection(s)
I20260925 01:27:24.393659 23704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:24.397890 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23704
I20260925 01:27:24.398701 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:43659
--local_ip_for_outbound_sockets=127.6.236.130
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=40027
--webserver_interface=127.6.236.130
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:27:24.399247 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.396748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:42481)
0925 01:27:24.397649 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.397952 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":301,"threads_started":1}
W20260925 01:27:24.400012 23872 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:45867 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:45867: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
I20260925 01:27:24.400931 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.400538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:33207)
0925 01:27:24.400711 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.400799 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:27:24.401799 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.401433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:53071)
0925 01:27:24.401581 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.401670 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 01:27:24.402029 23872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:27:24.711369 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:24.711723 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:24.711876 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:24.719265 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:24.719374 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:24.719504 23875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:24.729131 23875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:43659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=40027
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:24.730429 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:24.731730 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:24.744272 23880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:24.744302 23883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:24.744223 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:24.745143 23875 server_base.cc:1034] running on GCE node
I20260925 01:27:24.745690 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:24.746618 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:24.747807 23875 hybrid_clock.cc:648] HybridClock initialized: now 1790299644747756 us; error 41 us; skew 500 ppm
I20260925 01:27:24.748075 23875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:24 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:24.751121 23875 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:27:24.752544 23875 webserver.cc:460] Webserver started at http://127.6.236.130:40027/ using document root <none> and password file <none>
I20260925 01:27:24.753161 23875 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:24.753266 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:24.758467 23875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:24.761142 23890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.762492 23875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:24.762622 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "1a5baf361737425fbcafe5f3006249b1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "8746decc13602be7daa0aad8dfaec8cb"
server_key_iv: "36f1e6dd00028e250ceec4abb3922dac"
server_key_version: "encryptionkey@0"
I20260925 01:27:24.763528 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:24.787345 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:24.791244 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:24.791464 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:24.792372 23875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:24.793612 23875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:24.793700 23875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.793776 23875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:24.793828 23875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:24.824277 23875 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:43659
I20260925 01:27:24.824343 24002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:43659 every 8 connection(s)
I20260925 01:27:24.825696 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:24.830338 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.828562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:34541)
0925 01:27:24.829239 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.829474 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":247,"threads_started":1}
W20260925 01:27:24.830940 24003 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:45867 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:45867: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
I20260925 01:27:24.831103 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 23875
I20260925 01:27:24.831777 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.831463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:47415)
0925 01:27:24.831607 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.831674 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 01:27:24.831796 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:36043
--local_ip_for_outbound_sockets=127.6.236.131
--tserver_master_addrs=127.6.236.190:45867
--webserver_port=35735
--webserver_interface=127.6.236.131
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:27:24.832572 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:24.832288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:37897)
0925 01:27:24.832401 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:24.832472 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 01:27:24.832790 24003 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:27:25.148118 24006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:25.148384 24006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:25.148478 24006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:25.155684 24006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:25.155797 24006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:25.155916 24006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:25.165875 24006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:36043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=35735
--tserver_master_addrs=127.6.236.190:45867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:25.167337 24006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:25.168669 24006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:25.181335 24011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:25.181373 24012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:25.181396 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:25.182000 24006 server_base.cc:1034] running on GCE node
I20260925 01:27:25.182633 24006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:25.183548 24006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:25.184763 24006 hybrid_clock.cc:648] HybridClock initialized: now 1790299645184704 us; error 50 us; skew 500 ppm
I20260925 01:27:25.185056 24006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:25 dist-test-slave-s0b2 krb5kdc[22557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:25.188364 24006 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:27:25.190765 24006 webserver.cc:460] Webserver started at http://127.6.236.131:35735/ using document root <none> and password file <none>
I20260925 01:27:25.191629 24006 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:25.191761 24006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:25.197636 24006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:25.200246 24021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:25.201493 24006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:25.201656 24006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "9a43366176124a9eaf7013b0183940f1"
format_stamp: "Formatted at 2026-09-25 01:26:39 on dist-test-slave-s0b2"
server_key: "b932239ba47432ef178d8b2719b99541"
server_key_iv: "a1bb7ae21fdae6dab11c2ad6e88510a4"
server_key_version: "encryptionkey@0"
I20260925 01:27:25.202612 24006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:25.219338 24006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:25.223394 24006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:25.223640 24006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:25.224550 24006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:25.225798 24006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:25.225889 24006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:25.225991 24006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:25.226058 24006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:25.255373 24006 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:36043
I20260925 01:27:25.255424 24133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:36043 every 8 connection(s)
I20260925 01:27:25.256742 24006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:25.261318 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:25.259558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:38429)
0925 01:27:25.260275 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:25.260500 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":270,"threads_started":1}
W20260925 01:27:25.261936 24134 heartbeater.cc:644] Failed to heartbeat to 127.6.236.190:45867 (0 consecutive failures): Network error: Failed to ping master at 127.6.236.190:45867: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
I20260925 01:27:25.262699 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:25.262410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:57815)
0925 01:27:25.262535 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:25.262599 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 01:27:25.263422 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:25.263162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:57607)
0925 01:27:25.263261 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:25.263327 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 01:27:25.263633 24134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:27:25.263939 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24006
I20260925 01:27:25.403319 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:25.402569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:45443)
0925 01:27:25.403003 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:25.403106 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":209,"threads_started":1}
I20260925 01:27:25.834041 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:25.833305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:54905)
0925 01:27:25.833727 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:25.833822 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":195,"threads_started":1}
I20260925 01:27:26.264972 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:26.264173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:60703)
0925 01:27:26.264625 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:26.264753 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260925 01:27:26.404946 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:26.404210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:52997)
0925 01:27:26.404676 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:26.404774 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":241,"threads_started":1}
I20260925 01:27:26.835611 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:26.834914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:39259)
0925 01:27:26.835356 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:26.835438 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":215,"threads_started":1}
I20260925 01:27:27.266634 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:27.265875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:46443)
0925 01:27:27.266375 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:27.266468 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":218,"threads_started":1}
I20260925 01:27:27.406791 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:27.405903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:34243)
0925 01:27:27.406440 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:27.406535 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":278,"threads_started":1}
I20260925 01:27:27.837217 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:27.836465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:44753)
0925 01:27:27.836950 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:27.837040 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260925 01:27:28.268098 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:28.267436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:45947)
0925 01:27:28.267844 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:28.267930 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":213,"threads_started":1}
I20260925 01:27:28.408427 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:28.407704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:59625)
0925 01:27:28.408130 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:28.408213 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":209,"threads_started":1}
I20260925 01:27:28.838814 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:28.838051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:58525)
0925 01:27:28.838498 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:28.838580 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260925 01:27:29.269575 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:29.268895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:59009)
0925 01:27:29.269309 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:29.269390 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":190,"threads_started":1}
I20260925 01:27:29.409988 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:29.409282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:57001)
0925 01:27:29.409742 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:29.409823 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":234,"threads_started":1}
I20260925 01:27:29.840437 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:29.839709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:54311)
0925 01:27:29.840168 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:29.840269 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260925 01:27:30.271150 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:30.270433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:54457)
0925 01:27:30.270907 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:30.270988 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":234,"threads_started":1}
I20260925 01:27:30.411546 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:30.410825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:38077)
0925 01:27:30.411291 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:30.411384 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260925 01:27:30.841926 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:30.841260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:34565)
0925 01:27:30.841681 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:30.841772 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":209,"threads_started":1}
I20260925 01:27:31.272547 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:31.271874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:40323)
0925 01:27:31.272298 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:31.272388 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":209,"threads_started":1}
I20260925 01:27:31.413300 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:31.412492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:53993)
0925 01:27:31.413011 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:31.413089 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":238,"threads_started":1}
I20260925 01:27:31.843487 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:31.842771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:51111)
0925 01:27:31.843206 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:31.843284 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":178,"threads_started":1}
I20260925 01:27:32.274194 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:32.273426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:57943)
0925 01:27:32.273890 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:32.273973 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":196,"threads_started":1}
I20260925 01:27:32.414866 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:32.414158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:38713)
0925 01:27:32.414613 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:32.414691 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260925 01:27:32.845078 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:32.844330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:50599)
0925 01:27:32.844824 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:32.844907 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260925 01:27:33.275853 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:33.275133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:57281)
0925 01:27:33.275600 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:33.275683 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260925 01:27:33.416483 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:33.415727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:39849)
0925 01:27:33.416233 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:33.416313 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
I20260925 01:27:33.846741 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:33.845952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:55017)
0925 01:27:33.846459 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:33.846558 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":225,"threads_started":1}
I20260925 01:27:34.277441 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:34.276720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:39515)
0925 01:27:34.277186 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:34.277271 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":208,"threads_started":1}
I20260925 01:27:34.418133 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:34.417412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:49875)
0925 01:27:34.417878 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:34.417970 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260925 01:27:34.848362 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:34.847677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:41805)
0925 01:27:34.848118 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:34.848203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260925 01:27:35.279063 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:35.278375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:54697)
0925 01:27:35.278820 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:35.278906 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260925 01:27:35.419763 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:35.419041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:36471)
0925 01:27:35.419478 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:35.419569 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260925 01:27:35.849956 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:35.849249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:54421)
0925 01:27:35.849675 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:35.849777 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":202,"threads_started":1}
I20260925 01:27:36.280556 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:36.279879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:43683)
0925 01:27:36.280285 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:36.280380 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":198,"threads_started":1}
I20260925 01:27:36.421357 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:36.420643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:47711)
0925 01:27:36.421118 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:36.421197 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
I20260925 01:27:36.851635 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:36.850887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:43447)
0925 01:27:36.851349 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:36.851452 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":217,"threads_started":1}
I20260925 01:27:37.282049 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:37.281362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:41379)
0925 01:27:37.281810 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:37.281890 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":221,"threads_started":1}
I20260925 01:27:37.423094 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:37.422324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:50035)
0925 01:27:37.422805 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:37.422905 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":230,"threads_started":1}
I20260925 01:27:37.853432 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:37.852676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:46431)
0925 01:27:37.853147 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:37.853253 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260925 01:27:38.283704 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:38.282872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:38369)
0925 01:27:38.283391 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:38.283486 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260925 01:27:38.424759 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:38.424024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:52647)
0925 01:27:38.424493 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:38.424570 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260925 01:27:38.854944 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:38.854225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:52015)
0925 01:27:38.854663 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:38.854763 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260925 01:27:39.285228 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:39.284557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:43015)
0925 01:27:39.284971 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:39.285058 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260925 01:27:39.426637 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:39.425869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.129:35169)
0925 01:27:39.426352 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:39.426444 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260925 01:27:39.856696 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:39.855990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.130:46397)
0925 01:27:39.856412 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:39.856517 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":213,"threads_started":1}
I20260925 01:27:40.274273 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23704
I20260925 01:27:40.286950 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:40.286228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45867 (local address 127.6.236.131:48321)
0925 01:27:40.286690 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:40.286777 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.190:45867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260925 01:27:40.291812 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:40.275439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.6.236.129:40693)
0925 01:27:40.276189 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:40.276209 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:40.276258 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:27:40.278025 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:40.278035 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:40.278532 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:40.278841 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:40.280271 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.281471 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.282783 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.283426 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.283558 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:40.286105 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:40.286112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:40.286127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:40.286162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:40.288971 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.289608 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.289614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.289620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.289795 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.290086 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.290089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.290091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.290622 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:40.290855 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:40.291157 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 01:27:40.291536 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":413,"threads_started":1}
I20260925 01:27:40.293202 23867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:40.374888 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23704
I20260925 01:27:40.388993 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 23875
I20260925 01:27:40.406009 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:40.389878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.6.236.130:43659)
0925 01:27:40.390441 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:40.390456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:40.390494 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:27:40.391985 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:40.391994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:40.392429 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:40.392724 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:40.394038 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.395356 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.396559 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.397247 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.397378 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:40.399981 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:40.399990 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:40.400006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:40.400052 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:40.403042 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.403709 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.403715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.403723 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.403881 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.404166 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.404168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.404170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.404670 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:40.404888 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:40.405344 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 01:27:40.405724 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
I20260925 01:27:40.407481 23998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:40.488822 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 23875
I20260925 01:27:40.503847 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24006
I20260925 01:27:40.520802 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:40.504904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.6.236.131:36043)
0925 01:27:40.505552 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:40.505566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:40.505608 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:27:40.507354 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:40.507364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:40.507813 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:40.508109 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:40.509408 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.510663 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.511830 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:40.512458 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:40.512592 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:40.515080 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:40.515088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:40.515102 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:40.515146 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:40.517871 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.518481 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.518492 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.518498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.518662 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:40.519039 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:40.519042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:40.519044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:40.519538 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:40.519777 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:40.520118 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 01:27:40.520495 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":264,"threads_started":1}
I20260925 01:27:40.522090 24129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:40.602303 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24006
2026-09-25T01:27:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63872 ms)
[----------] 30 tests from SecurityITest (285189 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:27:40 dist-test-slave-s0b2 krb5kdc[24191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:27:40 dist-test-slave-s0b2 krb5kdc[24191](info): set up 2 sockets
Sep 25 01:27:40 dist-test-slave-s0b2 krb5kdc[24191](info): commencing operation
krb5kdc: starting...
W20260925 01:27:41.623111 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:27:41 dist-test-slave-s0b2 krb5kdc[24191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:27:41.855079 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45591
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:42.171051 24204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:42.171330 24204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:42.171424 24204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:42.178722 24204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:42.178840 24204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:42.178893 24204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:42.178947 24204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:42.178990 24204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:42.188835 24204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:42.190096 24204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:42.191437 24204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:42.202687 24209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:42.202659 24210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:42.202625 24212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:42.203648 24204 server_base.cc:1034] running on GCE node
I20260925 01:27:42.204351 24204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:42.205664 24204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:42.206867 24204 hybrid_clock.cc:648] HybridClock initialized: now 1790299662206819 us; error 33 us; skew 500 ppm
I20260925 01:27:42.207201 24204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:42 dist-test-slave-s0b2 krb5kdc[24191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299662, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:42.214124 24204 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:27:42.215628 24204 webserver.cc:460] Webserver started at http://127.6.236.190:38825/ using document root <none> and password file <none>
I20260925 01:27:42.216305 24204 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:42.216420 24204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:42.216702 24204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:42.219635 24204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "bf5697b647f54390bdfe0243fbf944ff"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "0fce7e3dac2d90dc14163a42a57dc6eb"
server_key_iv: "891e21f53d5acd5b434768a614e50eab"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.220358 24204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "bf5697b647f54390bdfe0243fbf944ff"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "0fce7e3dac2d90dc14163a42a57dc6eb"
server_key_iv: "891e21f53d5acd5b434768a614e50eab"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.225353 24204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:27:42.228624 24219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:42.229897 24204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:42.230053 24204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "bf5697b647f54390bdfe0243fbf944ff"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "0fce7e3dac2d90dc14163a42a57dc6eb"
server_key_iv: "891e21f53d5acd5b434768a614e50eab"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.230223 24204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:42.250128 24204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:42.253808 24204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:42.254063 24204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:42.270877 24204 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:45591
I20260925 01:27:42.270910 24270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:45591 every 8 connection(s)
I20260925 01:27:42.272200 24204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:42.275534 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24204
I20260925 01:27:42.275764 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:42.276163 24271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:42.276587 7090 external_mini_cluster.cc:1428] Setting key 25e454178607baf63e3c10688f57ecc1
I20260925 01:27:42.288493 24271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: Bootstrap starting.
Sep 25 01:27:42 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:42.291821 24271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:42.292965 24271 log.cc:824] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:42.295655 24271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: No bootstrap required, opened a new log
I20260925 01:27:42.298034 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:42.278748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.6.236.190:45591)
0925 01:27:42.279543 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:42.279555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:42.279598 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:27:42.281101 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:42.281122 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:42.281521 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:42.281888 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:42.283393 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.284646 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.285956 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.286686 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.286853 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:42.290967 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:42.290974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:42.290989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:42.291025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:42.294000 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:42.294761 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:42.294768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:42.294774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:42.294942 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:42.295321 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:42.295324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:42.295326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:42.295866 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:42.296171 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:42.296580 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 01:27:42.297006 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260925 01:27:42.303546 24271 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } }
I20260925 01:27:42.303900 24271 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:42.304004 24271 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5697b647f54390bdfe0243fbf944ff, State: Initialized, Role: FOLLOWER
I20260925 01:27:42.304517 24271 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } }
I20260925 01:27:42.304723 24271 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:42.304857 24271 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:42.304998 24271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:42.307116 24271 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } }
I20260925 01:27:42.307581 24271 leader_election.cc:302] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5697b647f54390bdfe0243fbf944ff; no voters:
I20260925 01:27:42.308120 24271 leader_election.cc:288] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:42.308248 24276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:42.309428 24276 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [term 1 LEADER]: Becoming Leader. State: Replica: bf5697b647f54390bdfe0243fbf944ff, State: Running, Role: LEADER
I20260925 01:27:42.309942 24276 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } }
I20260925 01:27:42.310184 24271 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:42.313050 24278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5697b647f54390bdfe0243fbf944ff. Latest consensus state: current_term: 1 leader_uuid: "bf5697b647f54390bdfe0243fbf944ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } } }
I20260925 01:27:42.313897 24278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:42.313086 24277 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5697b647f54390bdfe0243fbf944ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5697b647f54390bdfe0243fbf944ff" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 45591 } } }
I20260925 01:27:42.314293 24277 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:42.320762 24286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:42.325966 24286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:42.334761 24286 catalog_manager.cc:1347] Generated new cluster ID: 8e63c826da534541883346bdaa524451
I20260925 01:27:42.334865 24286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:42.357295 24286 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:42.358669 24286 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:42.373332 24286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: Generated new TSK 0
I20260925 01:27:42.374217 24286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:27:42.431689 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:42.747678 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:42.747947 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:42.748049 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:42.755502 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:42.755616 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:42.755736 24297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:42.765282 24297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:42.766463 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:42.767716 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:42.780078 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:42.780172 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:42.780323 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:42.780835 24297 server_base.cc:1034] running on GCE node
I20260925 01:27:42.781355 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:42.782181 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:42.783402 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790299662783358 us; error 34 us; skew 500 ppm
I20260925 01:27:42.783676 24297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:42 dist-test-slave-s0b2 krb5kdc[24191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299662, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:42.787424 24297 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:27:42.790421 24297 webserver.cc:460] Webserver started at http://127.6.236.129:43151/ using document root <none> and password file <none>
I20260925 01:27:42.791495 24297 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:42.791683 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:42.792274 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:42.797178 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "2870cf7862484554bf6ee3ae6e3d5997"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "648baf323ea2c86c7199985ddb161d03"
server_key_iv: "4e5a29c541fda184d313d81b734ab2d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.798327 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "2870cf7862484554bf6ee3ae6e3d5997"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "648baf323ea2c86c7199985ddb161d03"
server_key_iv: "4e5a29c541fda184d313d81b734ab2d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.804752 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260925 01:27:42.808461 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:42.810009 24297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:42.810163 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "2870cf7862484554bf6ee3ae6e3d5997"
format_stamp: "Formatted at 2026-09-25 01:27:42 on dist-test-slave-s0b2"
server_key: "648baf323ea2c86c7199985ddb161d03"
server_key_iv: "4e5a29c541fda184d313d81b734ab2d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:42.810472 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:42.825299 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:42.830523 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:42.830783 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:42.831815 24297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:42.833247 24297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:42.833321 24297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:42.833436 24297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:42.833501 24297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:42.862986 24297 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:43899
I20260925 01:27:42.863029 24424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:43899 every 8 connection(s)
I20260925 01:27:42.864198 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:42.874534 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24297
I20260925 01:27:42.874756 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:42.875590 7090 external_mini_cluster.cc:1428] Setting key 4ea185181488e2465bb3b277f13c3729
Sep 25 01:27:42 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299662, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:42.889456 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:42.866583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:35991 (local address 127.6.236.190:45591)
0925 01:27:42.867074 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:42.867080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:42.868111 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 01:27:42.869879 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:42.869885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:42.869970 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:42.870064 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:42.872053 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.873425 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.878327 (+ 4902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.878967 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.879094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:42.883503 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:42.883509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:42.883513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:42.883553 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:42.886371 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:42.887069 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:42.887075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:42.887078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:42.887218 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:42.887583 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:42.887587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:42.887590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:42.887921 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:42.888031 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:42.888885 (+ 854us) server_negotiation.cc:299] Negotiation successful
0925 01:27:42.889108 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":265,"threads_started":1}
I20260925 01:27:42.890399 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:42.867143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45591 (local address 127.6.236.129:35991)
0925 01:27:42.867957 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:42.867997 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:42.869619 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:42.870341 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:42.870352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:42.871034 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:42.871836 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:42.871861 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.873629 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.873635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:42.878110 (+ 4475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:42.878128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:42.879392 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:42.879399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:42.879736 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:42.880513 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:42.880548 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:42.883244 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:42.886501 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:42.886507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:42.886521 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:42.886948 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:42.887309 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:42.887312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:42.887314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:42.887488 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:42.888046 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:42.888054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:42.888508 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 01:27:42.888810 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 01:27:42.889281 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":321,"threads_started":1}
I20260925 01:27:42.892037 24425 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45591
I20260925 01:27:42.892333 24425 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:42.893026 24425 heartbeater.cc:511] Master 127.6.236.190:45591 requested a full tablet report, sending...
I20260925 01:27:42.895092 24236 ts_manager.cc:194] Registered new tserver with Master: 2870cf7862484554bf6ee3ae6e3d5997 (127.6.236.129:43899)
I20260925 01:27:42.896489 24236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:35991
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:27:42.929709 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:43.240957 24431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:43.241238 24431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:43.241364 24431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:43.248409 24431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:43.248514 24431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:43.248667 24431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:43.257992 24431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:43.259249 24431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:43.260521 24431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:43.273146 24437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:43.273422 24439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:43.273571 24436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:43.274205 24431 server_base.cc:1034] running on GCE node
I20260925 01:27:43.274806 24431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:43.275719 24431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:43.276962 24431 hybrid_clock.cc:648] HybridClock initialized: now 1790299663276914 us; error 43 us; skew 500 ppm
I20260925 01:27:43.277262 24431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:43 dist-test-slave-s0b2 krb5kdc[24191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:43.280200 24431 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:27:43.281582 24431 webserver.cc:460] Webserver started at http://127.6.236.130:42935/ using document root <none> and password file <none>
I20260925 01:27:43.282210 24431 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:43.282361 24431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:43.282655 24431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:43.285357 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "6c8667486b1a417a94236f9f69628ddb"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "4adc3e6bed5d57559e8a1490d0e50c02"
server_key_iv: "dc93104afa87eeb849b865a3f011e931"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.286067 24431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "6c8667486b1a417a94236f9f69628ddb"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "4adc3e6bed5d57559e8a1490d0e50c02"
server_key_iv: "dc93104afa87eeb849b865a3f011e931"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.290897 24431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:27:43.294265 24446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.295553 24431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:27:43.295687 24431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "6c8667486b1a417a94236f9f69628ddb"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "4adc3e6bed5d57559e8a1490d0e50c02"
server_key_iv: "dc93104afa87eeb849b865a3f011e931"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.295872 24431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:43.310765 24431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:43.314682 24431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:43.314952 24431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:43.315846 24431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:43.317068 24431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:43.317137 24431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.317250 24431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:43.317307 24431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.347286 24431 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:40843
I20260925 01:27:43.347308 24558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:40843 every 8 connection(s)
I20260925 01:27:43.348508 24431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:43.350543 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24431
I20260925 01:27:43.350704 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:27:43.351174 7090 external_mini_cluster.cc:1428] Setting key 60f61441c7777d7fb4a03ebafacf2628
Sep 25 01:27:43 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:43.367660 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.351009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:38383 (local address 127.6.236.190:45591)
0925 01:27:43.351227 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.351234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.352357 (+ 1123us) server_negotiation.cc:407] Connection header received
0925 01:27:43.354170 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.354176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.354285 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.354386 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:43.355899 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.356823 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.358302 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.358828 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.358939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.362224 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:43.362230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:43.362234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:43.362291 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:43.364735 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.365413 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.365418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.365420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.365549 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.366041 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.366044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.366046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.366356 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:43.366478 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.367228 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.367413 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:27:43.368943 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.351427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45591 (local address 127.6.236.130:38383)
0925 01:27:43.352196 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:43.352249 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:43.353906 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:43.354591 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:43.354599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:43.355153 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:43.355742 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.355760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.356950 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.356954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.358145 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.358157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.359007 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.359012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.359214 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.359824 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:43.359845 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:43.362008 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:43.364864 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:43.364874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:43.364891 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:43.365291 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:43.365698 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:43.365703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:43.365706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:43.365934 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:43.366486 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:43.366493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:43.366878 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 01:27:43.367142 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 01:27:43.367644 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":319,"threads_started":1}
I20260925 01:27:43.370645 24559 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45591
I20260925 01:27:43.370923 24559 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:43.371688 24559 heartbeater.cc:511] Master 127.6.236.190:45591 requested a full tablet report, sending...
I20260925 01:27:43.373232 24236 ts_manager.cc:194] Registered new tserver with Master: 6c8667486b1a417a94236f9f69628ddb (127.6.236.130:40843)
I20260925 01:27:43.374444 24236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:38383
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:27:43.404808 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:43.717337 24564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:43.717584 24564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:43.717689 24564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:43.724720 24564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:43.724820 24564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:43.724934 24564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:43.734309 24564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:43.735508 24564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:43.736708 24564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:43.750293 24569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:43.750341 24570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:43.750341 24572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:43.751329 24564 server_base.cc:1034] running on GCE node
I20260925 01:27:43.751891 24564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:43.752763 24564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:43.753949 24564 hybrid_clock.cc:648] HybridClock initialized: now 1790299663753900 us; error 43 us; skew 500 ppm
I20260925 01:27:43.754274 24564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:43 dist-test-slave-s0b2 krb5kdc[24191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:43.757104 24564 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:27:43.758512 24564 webserver.cc:460] Webserver started at http://127.6.236.131:32985/ using document root <none> and password file <none>
I20260925 01:27:43.759132 24564 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:43.759250 24564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:43.759527 24564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:43.762566 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "5aac6483c20c41e0b8e2fd24b89801c1"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "21dd20945211d841f145bb558f99f7b3"
server_key_iv: "03912c178ffd405136b6d325d9d24089"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.763319 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "5aac6483c20c41e0b8e2fd24b89801c1"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "21dd20945211d841f145bb558f99f7b3"
server_key_iv: "03912c178ffd405136b6d325d9d24089"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.768193 24564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:27:43.771477 24579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.772734 24564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:43.772886 24564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "5aac6483c20c41e0b8e2fd24b89801c1"
format_stamp: "Formatted at 2026-09-25 01:27:43 on dist-test-slave-s0b2"
server_key: "21dd20945211d841f145bb558f99f7b3"
server_key_iv: "03912c178ffd405136b6d325d9d24089"
server_key_version: "encryptionkey@0"
I20260925 01:27:43.773053 24564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:43.789271 24564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:43.792927 24564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:43.793175 24564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:43.794034 24564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:43.795317 24564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:43.795392 24564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.795511 24564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:43.795567 24564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:43.827842 24564 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:38643
I20260925 01:27:43.827889 24691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:38643 every 8 connection(s)
I20260925 01:27:43.829082 24564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:43.835860 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24564
I20260925 01:27:43.836009 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:27:43.836418 7090 external_mini_cluster.cc:1428] Setting key 0bf70abe783bf26bdb6f917fa5b3dd99
Sep 25 01:27:43 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299663, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:43.848553 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.831525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:33835 (local address 127.6.236.190:45591)
0925 01:27:43.831713 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.831717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.832818 (+ 1101us) server_negotiation.cc:407] Connection header received
0925 01:27:43.834712 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.834717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.834802 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.834886 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:43.836354 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.837182 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.839258 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.839754 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.839868 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.843066 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:43.843071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:43.843074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:43.843106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:43.845565 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.846351 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.846357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.846361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.846484 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.846887 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.846893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.846896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.847182 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:43.847318 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.847955 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.848180 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:27:43.849447 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.831981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:45591 (local address 127.6.236.131:33835)
0925 01:27:43.832648 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:43.832692 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:43.834457 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:43.835121 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:43.835129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:43.835596 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:43.836190 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.836213 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.837366 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.837371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.839115 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.839130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.839965 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.839973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.840211 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.840808 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:43.840829 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:43.842877 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:43.845693 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:43.845725 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:43.845742 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:43.846184 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:43.846580 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:43.846585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:43.846589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:43.846777 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:43.847305 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:43.847313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:43.847616 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 01:27:43.847848 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 01:27:43.848331 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
I20260925 01:27:43.851586 24692 heartbeater.cc:347] Connected to a master server at 127.6.236.190:45591
I20260925 01:27:43.851929 24692 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:43.852864 24692 heartbeater.cc:511] Master 127.6.236.190:45591 requested a full tablet report, sending...
I20260925 01:27:43.854179 24236 ts_manager.cc:194] Registered new tserver with Master: 5aac6483c20c41e0b8e2fd24b89801c1 (127.6.236.131:38643)
I20260925 01:27:43.854979 24236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:33835
I20260925 01:27:43.864580 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:43.881306 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.869822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.6.236.190:45591)
0925 01:27:43.870020 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.870026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.870449 (+ 423us) server_negotiation.cc:407] Connection header received
0925 01:27:43.870627 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.870630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.870693 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.870766 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:43.872053 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.872758 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.874006 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.874448 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.874536 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.876842 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:43.876846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:43.876850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:43.876878 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:43.879285 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.879848 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.879851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.879853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.879966 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:43.880225 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:43.880227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:43.880229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:43.880503 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:43.880675 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.880939 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.881100 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:27:43.884980 24236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:43.891642 24236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:43.899371 24425 heartbeater.cc:503] Master 127.6.236.190:45591 was elected leader, sending a full tablet report...
I20260925 01:27:43.917773 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.903594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43899 (local address 127.0.0.1:44342)
0925 01:27:43.908735 (+ 5141us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:43.908752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:43.908870 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:43.909706 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:43.909712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:43.909737 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:43.910158 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.910171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.912207 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.912212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.914740 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.914754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.916797 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.916803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.916991 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.917045 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 01:27:43.917158 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:27:43.917334 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5017,"mutex_wait_us":143,"thread_start_us":4886,"threads_started":1}
I20260925 01:27:43.918746 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.903677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.6.236.129:43899)
0925 01:27:43.908369 (+ 4692us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.908376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.908979 (+ 603us) server_negotiation.cc:407] Connection header received
0925 01:27:43.909067 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.909074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.909283 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.909460 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:43.910417 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.912050 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.914909 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.916310 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.916432 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.916651 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.918263 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.918431 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4533,"thread_start_us":4377,"threads_started":1}
I20260925 01:27:43.919772 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.904089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.6.236.131:38643)
0925 01:27:43.904652 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.904660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.907847 (+ 3187us) server_negotiation.cc:407] Connection header received
0925 01:27:43.908245 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.908253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.908499 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.908656 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:43.909573 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.912191 (+ 2618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.916065 (+ 3874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.917379 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.917469 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.917674 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.918429 (+ 755us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.918573 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
I20260925 01:27:43.920598 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.903586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:38643 (local address 127.0.0.1:51902)
0925 01:27:43.907714 (+ 4128us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:43.907754 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:43.907996 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:43.908976 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:43.908982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:43.909008 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:43.909381 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.909395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.912373 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.912378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.915895 (+ 3517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.915910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.917865 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.917871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.918222 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.918274 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 01:27:43.920090 (+ 1816us) client_negotiation.cc:240] Negotiation successful
0925 01:27:43.920228 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3910,"mutex_wait_us":21,"thread_start_us":3778,"threads_started":1}
I20260925 01:27:43.922999 24360 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 (DEFAULT_TABLE table=test-table [id=69cdb0e18cfd48feaa778216dcd45313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:43.923594 24627 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 (DEFAULT_TABLE table=test-table [id=69cdb0e18cfd48feaa778216dcd45313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:43.924427 24360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc361cdd7294d808b8bdf1c01b00937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:43.924893 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc361cdd7294d808b8bdf1c01b00937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:43.926441 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.908815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:40843 (local address 127.0.0.1:56348)
0925 01:27:43.917976 (+ 9161us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:43.917991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:43.918080 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:43.920122 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:43.920127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:43.920150 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:43.920320 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.920332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.922093 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.922098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.923833 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:43.923848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.925706 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.925712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:43.925882 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.925915 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:27:43.926014 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 01:27:43.926145 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9083,"thread_start_us":10230,"threads_started":1}
I20260925 01:27:43.926627 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:43.909049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.6.236.130:40843)
0925 01:27:43.919541 (+ 10492us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:43.919550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:43.919577 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:27:43.919677 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:43.919684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:43.919919 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:43.920497 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:43.920718 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.921951 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.924008 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:43.925506 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:43.925683 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:43.925870 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:43.926064 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 01:27:43.926214 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10344,"thread_start_us":10203,"threads_started":1}
I20260925 01:27:43.929996 24494 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 (DEFAULT_TABLE table=test-table [id=69cdb0e18cfd48feaa778216dcd45313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:43.931344 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc361cdd7294d808b8bdf1c01b00937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:43.935842 24710 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Bootstrap starting.
I20260925 01:27:43.938784 24711 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Bootstrap starting.
I20260925 01:27:43.939049 24710 tablet_bootstrap.cc:654] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:43.940376 24710 log.cc:824] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:43.942304 24711 tablet_bootstrap.cc:654] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:43.943342 24711 log.cc:824] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:43.945024 24710 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: No bootstrap required, opened a new log
I20260925 01:27:43.945330 24710 ts_tablet_manager.cc:1397] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260925 01:27:43.947770 24714 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Bootstrap starting.
I20260925 01:27:43.948799 24711 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: No bootstrap required, opened a new log
I20260925 01:27:43.949051 24711 ts_tablet_manager.cc:1397] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 01:27:43.950625 24714 tablet_bootstrap.cc:654] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:43.952023 24714 log.cc:824] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:43.954231 24714 tablet_bootstrap.cc:492] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: No bootstrap required, opened a new log
I20260925 01:27:43.954493 24714 ts_tablet_manager.cc:1397] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 01:27:43.955590 24710 raft_consensus.cc:348] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.956090 24710 raft_consensus.cc:374] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:43.956209 24710 raft_consensus.cc:729] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aac6483c20c41e0b8e2fd24b89801c1, State: Initialized, Role: FOLLOWER
I20260925 01:27:43.956861 24710 consensus_queue.cc:260] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.958016 24692 heartbeater.cc:503] Master 127.6.236.190:45591 was elected leader, sending a full tablet report...
I20260925 01:27:43.958626 24710 ts_tablet_manager.cc:1428] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 01:27:43.958877 24711 raft_consensus.cc:348] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.959424 24711 raft_consensus.cc:374] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:43.959537 24711 raft_consensus.cc:729] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2870cf7862484554bf6ee3ae6e3d5997, State: Initialized, Role: FOLLOWER
I20260925 01:27:43.960081 24711 consensus_queue.cc:260] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.961637 24711 ts_tablet_manager.cc:1428] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 01:27:43.964670 24714 raft_consensus.cc:348] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.965090 24714 raft_consensus.cc:374] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:43.965204 24714 raft_consensus.cc:729] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8667486b1a417a94236f9f69628ddb, State: Initialized, Role: FOLLOWER
I20260925 01:27:43.965952 24714 consensus_queue.cc:260] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:43.966974 24559 heartbeater.cc:503] Master 127.6.236.190:45591 was elected leader, sending a full tablet report...
I20260925 01:27:43.967516 24714 ts_tablet_manager.cc:1428] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
W20260925 01:27:44.080523 24693 tablet.cc:2369] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:44.100363 24560 tablet.cc:2369] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:44.116401 24426 tablet.cc:2369] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:44.177621 24716 raft_consensus.cc:482] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:44.177909 24716 raft_consensus.cc:504] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:44.179555 24716 leader_election.cc:288] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2870cf7862484554bf6ee3ae6e3d5997 (127.6.236.129:43899), 6c8667486b1a417a94236f9f69628ddb (127.6.236.130:40843)
I20260925 01:27:44.185181 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.179920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43899 (local address 127.6.236.131:38631)
0925 01:27:44.180124 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:44.180144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:44.180277 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:44.180922 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:44.180926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:44.180957 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:44.181129 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:44.181138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.182127 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.182133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:44.183445 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:44.183457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.184504 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.184509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:44.184683 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.184709 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:27:44.184786 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:27:44.184909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 01:27:44.185177 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.179990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:38631 (local address 127.6.236.129:43899)
0925 01:27:44.180161 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.180166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.180189 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:27:44.180713 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.180718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.180796 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.180886 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:44.181273 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.182003 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.183602 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.184368 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.184510 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.184581 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.184828 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.184950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:27:44.185719 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.180540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50501 (local address 127.6.236.130:40843)
0925 01:27:44.180720 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.180725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.181060 (+ 335us) server_negotiation.cc:407] Connection header received
0925 01:27:44.181220 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.181224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.181284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.181354 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:44.181847 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.182538 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.184141 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.184885 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.185027 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.185095 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.185398 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.185521 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:27:44.186383 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.180322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:40843 (local address 127.6.236.131:50501)
0925 01:27:44.180953 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:44.180970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:44.181090 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:44.181384 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:44.181388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:44.181410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:44.181706 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:44.181715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.182682 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.182685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:44.183993 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:44.184005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.185074 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.185079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:44.185215 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.185249 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:27:44.185920 (+ 671us) client_negotiation.cc:240] Negotiation successful
0925 01:27:44.186044 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":370,"threads_started":1}
I20260925 01:27:44.186332 24380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc361cdd7294d808b8bdf1c01b00937" candidate_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2870cf7862484554bf6ee3ae6e3d5997" is_pre_election: true
I20260925 01:27:44.186913 24380 raft_consensus.cc:2454] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aac6483c20c41e0b8e2fd24b89801c1 in term 0.
I20260925 01:27:44.187378 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc361cdd7294d808b8bdf1c01b00937" candidate_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8667486b1a417a94236f9f69628ddb" is_pre_election: true
I20260925 01:27:44.187563 24582 leader_election.cc:302] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2870cf7862484554bf6ee3ae6e3d5997, 5aac6483c20c41e0b8e2fd24b89801c1; no voters:
I20260925 01:27:44.187819 24514 raft_consensus.cc:2454] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aac6483c20c41e0b8e2fd24b89801c1 in term 0.
I20260925 01:27:44.187924 24716 raft_consensus.cc:2781] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:44.188054 24716 raft_consensus.cc:482] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:44.188139 24716 raft_consensus.cc:3037] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:44.190274 24716 raft_consensus.cc:504] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:44.190806 24716 leader_election.cc:288] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [CANDIDATE]: Term 1 election: Requested vote from peers 2870cf7862484554bf6ee3ae6e3d5997 (127.6.236.129:43899), 6c8667486b1a417a94236f9f69628ddb (127.6.236.130:40843)
I20260925 01:27:44.191295 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc361cdd7294d808b8bdf1c01b00937" candidate_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8667486b1a417a94236f9f69628ddb"
I20260925 01:27:44.191296 24380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc361cdd7294d808b8bdf1c01b00937" candidate_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2870cf7862484554bf6ee3ae6e3d5997"
I20260925 01:27:44.191507 24514 raft_consensus.cc:3037] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:44.191504 24380 raft_consensus.cc:3037] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:44.194638 24380 raft_consensus.cc:2454] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aac6483c20c41e0b8e2fd24b89801c1 in term 1.
I20260925 01:27:44.194633 24514 raft_consensus.cc:2454] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aac6483c20c41e0b8e2fd24b89801c1 in term 1.
I20260925 01:27:44.195000 24582 leader_election.cc:302] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2870cf7862484554bf6ee3ae6e3d5997, 5aac6483c20c41e0b8e2fd24b89801c1; no voters:
I20260925 01:27:44.195235 24716 raft_consensus.cc:2781] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:44.195667 24716 raft_consensus.cc:686] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5aac6483c20c41e0b8e2fd24b89801c1, State: Running, Role: LEADER
I20260925 01:27:44.196167 24716 consensus_queue.cc:237] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } }
I20260925 01:27:44.200526 24236 catalog_manager.cc:5495] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aac6483c20c41e0b8e2fd24b89801c1 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aac6483c20c41e0b8e2fd24b89801c1" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 } health_report { overall_health: UNKNOWN } } }
I20260925 01:27:44.276284 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.270893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51906 (local address 127.6.236.131:38643)
0925 01:27:44.271056 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.271062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.271211 (+ 149us) server_negotiation.cc:407] Connection header received
0925 01:27:44.271426 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.271430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.271510 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.271639 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:44.272031 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.272734 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.274127 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.274581 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.274690 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.275201 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:44.275719 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:44.275853 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.275942 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.276025 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 01:27:44.284622 24514 raft_consensus.cc:1261] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 1 FOLLOWER]: Refusing update from remote peer 5aac6483c20c41e0b8e2fd24b89801c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:44.284771 24380 raft_consensus.cc:1261] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 1 FOLLOWER]: Refusing update from remote peer 5aac6483c20c41e0b8e2fd24b89801c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:44.285542 24716 consensus_queue.cc:1035] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8667486b1a417a94236f9f69628ddb" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:44.285852 24720 consensus_queue.cc:1035] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2870cf7862484554bf6ee3ae6e3d5997" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:44.296401 24723 mvcc.cc:205] Tried to move back new op lower bound from 7333067424899555328 to 7333067424551153664. Current Snapshot: MvccSnapshot[applied={T|T < 7333067424899555328}]
I20260925 01:27:44.297364 24725 mvcc.cc:205] Tried to move back new op lower bound from 7333067424899555328 to 7333067424551153664. Current Snapshot: MvccSnapshot[applied={T|T < 7333067424899555328}]
I20260925 01:27:44.302136 24724 mvcc.cc:205] Tried to move back new op lower bound from 7333067424899555328 to 7333067424551153664. Current Snapshot: MvccSnapshot[applied={T|T < 7333067424899555328}]
I20260925 01:27:44.311038 24235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:44.311311 24235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:44.314837 24235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5697b647f54390bdfe0243fbf944ff: Sending DeleteTablet for 3 replicas of tablet 5dc361cdd7294d808b8bdf1c01b00937
I20260925 01:27:44.315876 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:44 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:51902
I20260925 01:27:44.315992 24494 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:44 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:56348
I20260925 01:27:44.316758 24732 tablet_replica.cc:330] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: stopping tablet replica
I20260925 01:27:44.316862 24360 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc361cdd7294d808b8bdf1c01b00937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:44 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:44342
I20260925 01:27:44.317222 24732 raft_consensus.cc:2229] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:44.317994 24732 raft_consensus.cc:2258] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:44.318086 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24297
I20260925 01:27:44.320407 24732 ts_tablet_manager.cc:1905] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:44.326759 24733 tablet_replica.cc:330] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: stopping tablet replica
I20260925 01:27:44.327203 24733 raft_consensus.cc:2229] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:44.327718 24733 raft_consensus.cc:2258] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:44.329900 24733 ts_tablet_manager.cc:1905] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:44.334524 24734 tablet_replica.cc:330] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: stopping tablet replica
I20260925 01:27:44.334926 24734 raft_consensus.cc:2229] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:44.335315 24734 raft_consensus.cc:2258] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:44.337548 24732 ts_tablet_manager.cc:1918] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:44.337692 24732 log.cc:1196] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/5dc361cdd7294d808b8bdf1c01b00937
I20260925 01:27:44.338145 24732 ts_tablet_manager.cc:1939] T 5dc361cdd7294d808b8bdf1c01b00937 P 5aac6483c20c41e0b8e2fd24b89801c1: Deleting consensus metadata
I20260925 01:27:44.339651 24223 catalog_manager.cc:4841] TS 5aac6483c20c41e0b8e2fd24b89801c1 (127.6.236.131:38643): tablet 5dc361cdd7294d808b8bdf1c01b00937 (table test-table [id=69cdb0e18cfd48feaa778216dcd45313]) successfully deleted
I20260925 01:27:44.341085 24734 ts_tablet_manager.cc:1905] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:27:44 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:27:44.347317 24734 ts_tablet_manager.cc:1918] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:44.347498 24734 log.cc:1196] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/5dc361cdd7294d808b8bdf1c01b00937
I20260925 01:27:44.347987 24734 ts_tablet_manager.cc:1939] T 5dc361cdd7294d808b8bdf1c01b00937 P 2870cf7862484554bf6ee3ae6e3d5997: Deleting consensus metadata
I20260925 01:27:44.349684 24733 ts_tablet_manager.cc:1918] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:44.349855 24733 log.cc:1196] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/5dc361cdd7294d808b8bdf1c01b00937
I20260925 01:27:44.350414 24733 ts_tablet_manager.cc:1939] T 5dc361cdd7294d808b8bdf1c01b00937 P 6c8667486b1a417a94236f9f69628ddb: Deleting consensus metadata
I20260925 01:27:44.352203 24222 catalog_manager.cc:4841] TS 6c8667486b1a417a94236f9f69628ddb (127.6.236.130:40843): tablet 5dc361cdd7294d808b8bdf1c01b00937 (table test-table [id=69cdb0e18cfd48feaa778216dcd45313]) successfully deleted
I20260925 01:27:44.352480 24222 catalog_manager.cc:4841] TS 2870cf7862484554bf6ee3ae6e3d5997 (127.6.236.129:43899): tablet 5dc361cdd7294d808b8bdf1c01b00937 (table test-table [id=69cdb0e18cfd48feaa778216dcd45313]) successfully deleted
I20260925 01:27:44.357812 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.320994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.6.236.129:43899)
0925 01:27:44.321156 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.321162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.333848 (+ 12686us) server_negotiation.cc:407] Connection header received
0925 01:27:44.334041 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.334050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.334151 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.334369 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:44.336151 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.337226 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.339191 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.339772 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.339915 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.345186 (+ 5271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:44.345201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:44.345222 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:44.345280 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:44.354560 (+ 9280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.355414 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.355424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.355428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.355590 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.355990 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.355996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.355999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.356434 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:44.356737 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.357037 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.357267 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:27:44.358472 24420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:44.495358 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24297
I20260925 01:27:44.510824 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24431
Sep 25 01:27:44 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:27:44.525875 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.511650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.6.236.130:40843)
0925 01:27:44.511818 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.511824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.511904 (+ 80us) server_negotiation.cc:407] Connection header received
0925 01:27:44.512097 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.512102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.512169 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.512266 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:44.513556 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.514266 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.515493 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.515890 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.516008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.520196 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:44.520211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:44.520226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:44.520259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:44.523217 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.523915 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.523923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.523925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.524049 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.524406 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.524410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.524412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.524774 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:44.525038 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.525356 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.525556 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:27:44.526515 24554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:44.614763 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24431
I20260925 01:27:44.629534 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24564
Sep 25 01:27:44 dist-test-slave-s0b2 krb5kdc[24191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:27:44.645462 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:44.630315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.6.236.131:38643)
0925 01:27:44.630491 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:44.630498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:44.630539 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:27:44.630792 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:44.630798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:44.630884 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:44.630968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:44.632220 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.633020 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.634392 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:44.634822 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:44.634935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:44.639701 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:44.639713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:44.639728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:44.639764 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:44.642742 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.643386 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.643393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.643395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.643558 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:44.643932 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:44.643935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:44.643937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:44.644300 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:44.644601 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:44.644888 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 01:27:44.645095 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:27:44.646075 24687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:44.766103 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24564
I20260925 01:27:44.782352 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24204
I20260925 01:27:44.783479 24266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:44.891366 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24204
2026-09-25T01:27:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T01:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:44Z Disabled control of system clock
I20260925 01:27:44.962524 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42513
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:42513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:45.275314 24743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:45.275590 24743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:45.275714 24743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:45.282987 24743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:45.283093 24743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:45.283157 24743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:45.283241 24743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:45.283299 24743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:45.292333 24743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:42513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:45.293576 24743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:45.295408 24743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:45.306417 24749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:45.306563 24748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:45.306501 24751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:45.307405 24743 server_base.cc:1034] running on GCE node
I20260925 01:27:45.308174 24743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:45.309520 24743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:45.310721 24743 hybrid_clock.cc:648] HybridClock initialized: now 1790299665310662 us; error 53 us; skew 500 ppm
I20260925 01:27:45.311046 24743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:45.313099 24743 webserver.cc:460] Webserver started at http://127.6.236.190:40057/ using document root <none> and password file <none>
I20260925 01:27:45.313746 24743 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:45.313844 24743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:45.314134 24743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:45.316907 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1bc40af734074a91ab8135f1cb18839f"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "28edca9bd856ba61baf30a5c449488e9"
server_key_iv: "36a350a2bb509aff98af68827f4fc6d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.317655 24743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1bc40af734074a91ab8135f1cb18839f"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "28edca9bd856ba61baf30a5c449488e9"
server_key_iv: "36a350a2bb509aff98af68827f4fc6d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.322574 24743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:27:45.325742 24757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:45.326992 24743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:45.327149 24743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1bc40af734074a91ab8135f1cb18839f"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "28edca9bd856ba61baf30a5c449488e9"
server_key_iv: "36a350a2bb509aff98af68827f4fc6d2"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.327345 24743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:45.349710 24743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:45.350584 24743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:45.350857 24743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:45.368268 24743 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:42513
I20260925 01:27:45.368287 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:42513 every 8 connection(s)
I20260925 01:27:45.369606 24743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:45.371546 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24743
I20260925 01:27:45.371697 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:45.372172 7090 external_mini_cluster.cc:1428] Setting key 02c7e0b1f27c904b90d920766ebea2c3
I20260925 01:27:45.373657 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:45.387563 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:45.374031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.6.236.190:42513)
0925 01:27:45.375075 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:45.375090 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:45.375143 (+ 53us) server_negotiation.cc:407] Connection header received
0925 01:27:45.377010 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:45.377040 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:45.377676 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:45.378145 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:45.378739 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.380755 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.382359 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.383347 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.383536 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:45.384676 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:45.384688 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:45.384717 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:45.385494 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:45.385589 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:45.385615 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:45.385749 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:45.385906 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:27:45.386283 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":327,"threads_started":1}
I20260925 01:27:45.388377 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: Bootstrap starting.
I20260925 01:27:45.391527 24809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:45.392496 24809 log.cc:824] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:45.395246 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: No bootstrap required, opened a new log
I20260925 01:27:45.402779 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } }
I20260925 01:27:45.403124 24809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:45.403227 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc40af734074a91ab8135f1cb18839f, State: Initialized, Role: FOLLOWER
I20260925 01:27:45.403789 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } }
I20260925 01:27:45.404014 24809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:45.404141 24809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:45.404294 24809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:45.406508 24809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } }
I20260925 01:27:45.406984 24809 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc40af734074a91ab8135f1cb18839f; no voters:
I20260925 01:27:45.407500 24809 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:45.407630 24814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:45.408267 24814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [term 1 LEADER]: Becoming Leader. State: Replica: 1bc40af734074a91ab8135f1cb18839f, State: Running, Role: LEADER
I20260925 01:27:45.408740 24814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } }
I20260925 01:27:45.409407 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:45.411284 24816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc40af734074a91ab8135f1cb18839f. Latest consensus state: current_term: 1 leader_uuid: "1bc40af734074a91ab8135f1cb18839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } } }
I20260925 01:27:45.411698 24816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:45.412393 24815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc40af734074a91ab8135f1cb18839f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc40af734074a91ab8135f1cb18839f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42513 } } }
I20260925 01:27:45.412683 24815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:45.413403 24820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:45.418954 24820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:45.426890 24820 catalog_manager.cc:1347] Generated new cluster ID: 638f2fdb9b2b4c118bfa051310c43b9e
I20260925 01:27:45.427006 24820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:45.467288 24820 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:45.468559 24820 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:45.479765 24820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: Generated new TSK 0
I20260925 01:27:45.480543 24820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:27:45.489848 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--builtin_ntp_servers=127.6.236.148:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:45.809669 24833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:45.809966 24833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:45.810094 24833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:45.817494 24833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:45.817596 24833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:45.817716 24833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:45.826954 24833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:45.828161 24833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:45.829845 24833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:45.842487 24838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:45.842528 24839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:45.842423 24841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:45.843269 24833 server_base.cc:1034] running on GCE node
I20260925 01:27:45.843874 24833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:45.844749 24833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:45.845981 24833 hybrid_clock.cc:648] HybridClock initialized: now 1790299665845921 us; error 49 us; skew 500 ppm
I20260925 01:27:45.846361 24833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:45.848544 24833 webserver.cc:460] Webserver started at http://127.6.236.129:38129/ using document root <none> and password file <none>
I20260925 01:27:45.849217 24833 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:45.849316 24833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:45.849622 24833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:45.852466 24833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "a79f5d6a8d044c0c8237438a4b6897cf"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "52be961fd166290356ad73236ca7b7a2"
server_key_iv: "912569fdd8792d03cf36b7facf70b969"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.853221 24833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "a79f5d6a8d044c0c8237438a4b6897cf"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "52be961fd166290356ad73236ca7b7a2"
server_key_iv: "912569fdd8792d03cf36b7facf70b969"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.858161 24833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:27:45.861629 24847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:45.862939 24833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:27:45.863086 24833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "a79f5d6a8d044c0c8237438a4b6897cf"
format_stamp: "Formatted at 2026-09-25 01:27:45 on dist-test-slave-s0b2"
server_key: "52be961fd166290356ad73236ca7b7a2"
server_key_iv: "912569fdd8792d03cf36b7facf70b969"
server_key_version: "encryptionkey@0"
I20260925 01:27:45.863281 24833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:45.888197 24833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:45.889053 24833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:45.889349 24833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:45.890345 24833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:45.891829 24833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:45.891923 24833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:45.892025 24833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:45.892094 24833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:45.921033 24833 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:43521
I20260925 01:27:45.921103 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:43521 every 8 connection(s)
I20260925 01:27:45.922343 24833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:45.932380 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24833
I20260925 01:27:45.932539 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:45.932981 7090 external_mini_cluster.cc:1428] Setting key 7894bc35fb4c03297c875909468d9d88
I20260925 01:27:45.934136 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:45.924712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:46139 (local address 127.6.236.190:42513)
0925 01:27:45.925134 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:45.925138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:45.926272 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 01:27:45.928036 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:45.928041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:45.928125 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:45.928216 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:45.929253 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.929957 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.931485 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.931882 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.931990 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:45.933310 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:45.933315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:45.933318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:45.933336 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:45.933376 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:45.933388 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:45.933461 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:45.933672 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:27:45.933783 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260925 01:27:45.935063 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:45.925286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:42513 (local address 127.6.236.129:46139)
0925 01:27:45.926109 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:45.926150 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:45.927776 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:45.928454 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:45.928462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:45.928550 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:45.929090 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:45.929106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.930093 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.930097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:45.931348 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:45.931361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:45.932125 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:45.932133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:45.932350 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:45.932988 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:45.932990 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:45.933032 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:45.933130 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:45.933471 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:45.933482 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:45.933490 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:27:45.933647 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:27:45.934014 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":227,"threads_started":1}
I20260925 01:27:45.935652 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--builtin_ntp_servers=127.6.236.148:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:27:45.936483 24960 heartbeater.cc:347] Connected to a master server at 127.6.236.190:42513
I20260925 01:27:45.936755 24960 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:45.937439 24960 heartbeater.cc:511] Master 127.6.236.190:42513 requested a full tablet report, sending...
I20260925 01:27:45.939468 24774 ts_manager.cc:194] Registered new tserver with Master: a79f5d6a8d044c0c8237438a4b6897cf (127.6.236.129:43521)
I20260925 01:27:45.941385 24774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.129:46139
W20260925 01:27:46.248286 24964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:46.248579 24964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:46.248673 24964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:46.255859 24964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:46.255961 24964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:46.256073 24964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:46.265043 24964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:46.266309 24964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:46.268146 24964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:46.280097 24970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:46.280275 24972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:46.280674 24969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:46.281112 24964 server_base.cc:1034] running on GCE node
I20260925 01:27:46.281693 24964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:46.282634 24964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:46.283836 24964 hybrid_clock.cc:648] HybridClock initialized: now 1790299666283781 us; error 46 us; skew 500 ppm
I20260925 01:27:46.284125 24964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:46.286180 24964 webserver.cc:460] Webserver started at http://127.6.236.130:42293/ using document root <none> and password file <none>
I20260925 01:27:46.286859 24964 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:46.286959 24964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:46.287256 24964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:46.289997 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "5e58d13f76a0470c83c904234b5a0037"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "ffb0ccf90792fa31644ff959757e0e3e"
server_key_iv: "cf5ed787cd5b01f768798e62b53a40b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.290789 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "5e58d13f76a0470c83c904234b5a0037"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "ffb0ccf90792fa31644ff959757e0e3e"
server_key_iv: "cf5ed787cd5b01f768798e62b53a40b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.295581 24964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:27:46.298847 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.300076 24964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:46.300225 24964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "5e58d13f76a0470c83c904234b5a0037"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "ffb0ccf90792fa31644ff959757e0e3e"
server_key_iv: "cf5ed787cd5b01f768798e62b53a40b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.300416 24964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:46.320087 24964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:46.320851 24964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:46.321117 24964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:46.322027 24964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:46.323436 24964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:46.323531 24964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.323635 24964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:46.323704 24964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.351501 24964 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:42861
I20260925 01:27:46.351518 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:42861 every 8 connection(s)
I20260925 01:27:46.352725 24964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:46.358448 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 24964
I20260925 01:27:46.358608 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:27:46.359100 7090 external_mini_cluster.cc:1428] Setting key d59ae6d32db8d01b4e65d3735f542414
I20260925 01:27:46.362198 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--builtin_ntp_servers=127.6.236.148:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:27:46.365407 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.355121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:49361 (local address 127.6.236.190:42513)
0925 01:27:46.355301 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.355305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.356681 (+ 1376us) server_negotiation.cc:407] Connection header received
0925 01:27:46.358466 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.358472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.358561 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.358659 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:46.359788 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.360670 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.362854 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.363243 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.363338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.364753 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:46.364758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:46.364762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:46.364779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:46.364810 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:46.364822 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:46.364886 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.365089 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.365168 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:27:46.366722 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.355660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:42513 (local address 127.6.236.130:49361)
0925 01:27:46.356529 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.356565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.358171 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.358899 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.358908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.359002 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:46.359605 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.359623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.360818 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.360823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.362713 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.362729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.363457 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.363462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.363733 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.364414 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:46.364418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:46.364467 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:46.364570 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:46.364901 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:46.364911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:46.364917 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.365081 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.365465 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260925 01:27:46.368275 25091 heartbeater.cc:347] Connected to a master server at 127.6.236.190:42513
I20260925 01:27:46.368573 25091 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:46.369410 25091 heartbeater.cc:511] Master 127.6.236.190:42513 requested a full tablet report, sending...
I20260925 01:27:46.370878 24774 ts_manager.cc:194] Registered new tserver with Master: 5e58d13f76a0470c83c904234b5a0037 (127.6.236.130:42861)
I20260925 01:27:46.371891 24774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.130:49361
W20260925 01:27:46.682489 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:46.682756 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:46.682840 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:46.689877 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:46.689970 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:46.690069 25094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:46.698956 25094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:46.700219 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:46.701908 25094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:46.714538 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:46.714476 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:46.715085 25094 server_base.cc:1034] running on GCE node
W20260925 01:27:46.715173 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:46.715761 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:46.716638 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:46.717871 25094 hybrid_clock.cc:648] HybridClock initialized: now 1790299666717814 us; error 48 us; skew 500 ppm
I20260925 01:27:46.718197 25094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:46.720285 25094 webserver.cc:460] Webserver started at http://127.6.236.131:35547/ using document root <none> and password file <none>
I20260925 01:27:46.720942 25094 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:46.721038 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:46.721341 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:46.724087 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "ecc35bd69967402b9eeb700614baf46b"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "f4f42ea7c01e982025b335d1e22d30d1"
server_key_iv: "3b8717b402b07bcfda82a5d35ee0dbc3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.724964 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "ecc35bd69967402b9eeb700614baf46b"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "f4f42ea7c01e982025b335d1e22d30d1"
server_key_iv: "3b8717b402b07bcfda82a5d35ee0dbc3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.729842 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 01:27:46.733207 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.734493 25094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:46.734634 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "ecc35bd69967402b9eeb700614baf46b"
format_stamp: "Formatted at 2026-09-25 01:27:46 on dist-test-slave-s0b2"
server_key: "f4f42ea7c01e982025b335d1e22d30d1"
server_key_iv: "3b8717b402b07bcfda82a5d35ee0dbc3"
server_key_version: "encryptionkey@0"
I20260925 01:27:46.734828 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:46.759130 25094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:46.759943 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:46.760216 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:46.761133 25094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:46.762580 25094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:46.762673 25094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.762780 25094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:46.762849 25094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:46.792424 25094 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:35983
I20260925 01:27:46.792478 25220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:35983 every 8 connection(s)
I20260925 01:27:46.793632 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:46.799010 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25094
I20260925 01:27:46.799158 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:27:46.799698 7090 external_mini_cluster.cc:1428] Setting key dede048dea34b20a0f991ffbc8071afb
I20260925 01:27:46.805837 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.796035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:60111 (local address 127.6.236.190:42513)
0925 01:27:46.796209 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.796214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.797543 (+ 1329us) server_negotiation.cc:407] Connection header received
0925 01:27:46.799399 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.799405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.799493 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.799602 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:46.800900 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.801849 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.803532 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.803954 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.804047 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.805221 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:46.805225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:46.805229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:46.805246 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:46.805277 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:46.805289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:46.805351 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.805505 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.805579 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:27:46.806816 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.796512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:42513 (local address 127.6.236.131:60111)
0925 01:27:46.797396 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.797439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.799073 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.799834 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.799844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.799941 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:46.800633 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.800659 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.801991 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.801996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.803399 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.803415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.804122 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.804127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.804329 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.804920 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:46.804923 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:46.804959 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:46.805036 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:46.805355 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:46.805360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:46.805365 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.805491 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.805761 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":246,"threads_started":1}
I20260925 01:27:46.808035 25221 heartbeater.cc:347] Connected to a master server at 127.6.236.190:42513
I20260925 01:27:46.808297 25221 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:46.808961 25221 heartbeater.cc:511] Master 127.6.236.190:42513 requested a full tablet report, sending...
I20260925 01:27:46.810206 24774 ts_manager.cc:194] Registered new tserver with Master: ecc35bd69967402b9eeb700614baf46b (127.6.236.131:35983)
I20260925 01:27:46.811070 24774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.131:60111
I20260925 01:27:46.814406 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:46.823441 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.818140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.6.236.190:42513)
0925 01:27:46.818316 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.818320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.818699 (+ 379us) server_negotiation.cc:407] Connection header received
0925 01:27:46.818809 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.818812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.818881 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.818958 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:46.819403 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.820094 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.821495 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.821909 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.821998 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.822942 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:46.822947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:46.822950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:46.822967 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:46.822997 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:46.823009 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:46.823090 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.823164 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.823234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:27:46.827011 24774 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:46.833957 24774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:46.856400 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.847507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:35983 (local address 127.0.0.1:55176)
0925 01:27:46.848058 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.848076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.848189 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.849465 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.849473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.849503 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:46.849856 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.849869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.851376 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.851381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.854057 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.854072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.855465 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.855471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.855636 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.855722 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.855833 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.856009 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":262,"thread_start_us":891,"threads_started":1}
I20260925 01:27:46.856992 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.846005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.6.236.129:43521)
0925 01:27:46.847365 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.847374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.848783 (+ 1409us) server_negotiation.cc:407] Connection header received
0925 01:27:46.848890 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.848898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.849169 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.849355 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:46.850317 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.851805 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.854000 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.855301 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.855428 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.855670 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.856493 (+ 823us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.856660 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":1040,"threads_started":1}
I20260925 01:27:46.857127 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.845195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42861 (local address 127.0.0.1:34766)
0925 01:27:46.846593 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.846635 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.846788 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.849955 (+ 3167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.849960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.849991 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:46.850424 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.850436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.852193 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.852199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.854396 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.854409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.856088 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.856095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.856273 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.856301 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.856364 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.856814 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":52,"thread_start_us":1006,"threads_started":1}
I20260925 01:27:46.857543 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.848093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.6.236.131:35983)
0925 01:27:46.848876 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.848884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.848918 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:27:46.849019 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.849027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.849261 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.849440 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:46.850065 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.851255 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.854325 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.855266 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.856843 (+ 1577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.857037 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.857103 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.857247 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":440,"threads_started":1}
I20260925 01:27:46.857853 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.845475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43521 (local address 127.0.0.1:37438)
0925 01:27:46.847763 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.847780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.847905 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.849621 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.849626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.849655 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:46.850111 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.850125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.851964 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.851969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.853834 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.853847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.855815 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.855821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.856004 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.856041 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.856120 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.856247 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2150,"mutex_wait_us":80,"thread_start_us":2026,"threads_started":1}
I20260925 01:27:46.858067 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.845405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.6.236.130:42861)
0925 01:27:46.849373 (+ 3968us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.849387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.849413 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:27:46.849514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.849521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.849772 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.849959 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:46.850585 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.852031 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.854561 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.855881 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.857295 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.857532 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.857589 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.857724 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3799,"thread_start_us":3686,"threads_started":1}
I20260925 01:27:46.861143 24895 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd776f75734fceba5011b38c56d8cc (DEFAULT_TABLE table=test-table [id=36c164e412e84cf1b0de85f1323cd5d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:46.861155 25156 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd776f75734fceba5011b38c56d8cc (DEFAULT_TABLE table=test-table [id=36c164e412e84cf1b0de85f1323cd5d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:46.861850 25026 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd776f75734fceba5011b38c56d8cc (DEFAULT_TABLE table=test-table [id=36c164e412e84cf1b0de85f1323cd5d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:46.862495 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04fd776f75734fceba5011b38c56d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:46.862598 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04fd776f75734fceba5011b38c56d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:46.863086 25026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04fd776f75734fceba5011b38c56d8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:46.873227 25239 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Bootstrap starting.
I20260925 01:27:46.877552 25239 tablet_bootstrap.cc:654] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:46.878412 25240 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Bootstrap starting.
I20260925 01:27:46.878839 25239 log.cc:824] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:46.879299 25241 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Bootstrap starting.
I20260925 01:27:46.882131 25240 tablet_bootstrap.cc:654] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:46.882476 25241 tablet_bootstrap.cc:654] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:46.882561 25239 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: No bootstrap required, opened a new log
I20260925 01:27:46.883067 25239 ts_tablet_manager.cc:1397] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:27:46.883311 25240 log.cc:824] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:46.883584 25241 log.cc:824] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:46.885965 25241 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: No bootstrap required, opened a new log
I20260925 01:27:46.886152 25240 tablet_bootstrap.cc:492] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: No bootstrap required, opened a new log
I20260925 01:27:46.886188 25241 ts_tablet_manager.cc:1397] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:27:46.886416 25240 ts_tablet_manager.cc:1397] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 01:27:46.892681 25241 raft_consensus.cc:348] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.893034 25241 raft_consensus.cc:374] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:46.893388 25241 raft_consensus.cc:729] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc35bd69967402b9eeb700614baf46b, State: Initialized, Role: FOLLOWER
I20260925 01:27:46.893484 25239 raft_consensus.cc:348] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.893993 25239 raft_consensus.cc:374] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:46.894093 25239 raft_consensus.cc:729] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79f5d6a8d044c0c8237438a4b6897cf, State: Initialized, Role: FOLLOWER
I20260925 01:27:46.894011 25241 consensus_queue.cc:260] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.894748 25239 consensus_queue.cc:260] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.895428 25221 heartbeater.cc:503] Master 127.6.236.190:42513 was elected leader, sending a full tablet report...
I20260925 01:27:46.896070 25241 ts_tablet_manager.cc:1428] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260925 01:27:46.896533 25240 raft_consensus.cc:348] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.897055 25240 raft_consensus.cc:374] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:46.897262 25240 raft_consensus.cc:729] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e58d13f76a0470c83c904234b5a0037, State: Initialized, Role: FOLLOWER
I20260925 01:27:46.897418 24960 heartbeater.cc:503] Master 127.6.236.190:42513 was elected leader, sending a full tablet report...
I20260925 01:27:46.898067 25239 ts_tablet_manager.cc:1428] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260925 01:27:46.897881 25240 consensus_queue.cc:260] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.899163 25091 heartbeater.cc:503] Master 127.6.236.190:42513 was elected leader, sending a full tablet report...
I20260925 01:27:46.899569 25240 ts_tablet_manager.cc:1428] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260925 01:27:46.924328 24961 tablet.cc:2369] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:46.951534 25245 raft_consensus.cc:482] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:46.951828 25245 raft_consensus.cc:504] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.953475 25245 leader_election.cc:288] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e58d13f76a0470c83c904234b5a0037 (127.6.236.130:42861), a79f5d6a8d044c0c8237438a4b6897cf (127.6.236.129:43521)
I20260925 01:27:46.959409 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.953700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:42861 (local address 127.6.236.131:50635)
0925 01:27:46.953889 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.953909 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.954027 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.954475 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.954480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.954528 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:46.954743 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.954755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.955880 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.955884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.957684 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.957712 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.958904 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.958908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.959022 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.959046 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.959116 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.959209 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:27:46.959517 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.953920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:50635 (local address 127.6.236.130:42861)
0925 01:27:46.954093 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.954098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.954118 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:27:46.954198 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.954204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.954309 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.954409 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:46.954882 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.955754 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.957850 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.958779 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.958931 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.959002 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.959160 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.959282 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:27:46.960254 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.953899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:43521 (local address 127.6.236.131:54679)
0925 01:27:46.954658 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:46.954679 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:46.954794 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:46.955117 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:46.955120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:46.955138 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:46.955411 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.955418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.956458 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.956463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.958180 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:46.958190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.959757 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.959763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:46.959890 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.959912 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:27:46.959982 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:27:46.960061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":409,"threads_started":1}
I20260925 01:27:46.960410 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:46.954093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:54679 (local address 127.6.236.129:43521)
0925 01:27:46.954297 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:46.954302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:46.954776 (+ 474us) server_negotiation.cc:407] Connection header received
0925 01:27:46.954924 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:46.954929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:46.955015 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:46.955108 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:46.955553 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.956312 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.958380 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:46.959491 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:46.959634 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:46.959697 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:46.960042 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 01:27:46.960166 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:27:46.960454 25046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd776f75734fceba5011b38c56d8cc" candidate_uuid: "ecc35bd69967402b9eeb700614baf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e58d13f76a0470c83c904234b5a0037" is_pre_election: true
I20260925 01:27:46.960971 25046 raft_consensus.cc:2454] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc35bd69967402b9eeb700614baf46b in term 0.
I20260925 01:27:46.961184 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd776f75734fceba5011b38c56d8cc" candidate_uuid: "ecc35bd69967402b9eeb700614baf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" is_pre_election: true
I20260925 01:27:46.961594 25110 leader_election.cc:302] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e58d13f76a0470c83c904234b5a0037, ecc35bd69967402b9eeb700614baf46b; no voters:
I20260925 01:27:46.961671 24915 raft_consensus.cc:2454] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc35bd69967402b9eeb700614baf46b in term 0.
I20260925 01:27:46.961938 25245 raft_consensus.cc:2781] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:46.962080 25245 raft_consensus.cc:482] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:46.962186 25245 raft_consensus.cc:3037] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:46.964282 25245 raft_consensus.cc:504] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.964797 25245 leader_election.cc:288] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [CANDIDATE]: Term 1 election: Requested vote from peers 5e58d13f76a0470c83c904234b5a0037 (127.6.236.130:42861), a79f5d6a8d044c0c8237438a4b6897cf (127.6.236.129:43521)
I20260925 01:27:46.965175 25046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd776f75734fceba5011b38c56d8cc" candidate_uuid: "ecc35bd69967402b9eeb700614baf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e58d13f76a0470c83c904234b5a0037"
I20260925 01:27:46.965247 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd776f75734fceba5011b38c56d8cc" candidate_uuid: "ecc35bd69967402b9eeb700614baf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79f5d6a8d044c0c8237438a4b6897cf"
I20260925 01:27:46.965337 25046 raft_consensus.cc:3037] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:46.965436 24915 raft_consensus.cc:3037] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:46.967950 25046 raft_consensus.cc:2454] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc35bd69967402b9eeb700614baf46b in term 1.
I20260925 01:27:46.968032 24915 raft_consensus.cc:2454] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc35bd69967402b9eeb700614baf46b in term 1.
I20260925 01:27:46.968353 25110 leader_election.cc:302] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e58d13f76a0470c83c904234b5a0037, ecc35bd69967402b9eeb700614baf46b; no voters:
I20260925 01:27:46.968649 25245 raft_consensus.cc:2781] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:46.969018 25245 raft_consensus.cc:686] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 1 LEADER]: Becoming Leader. State: Replica: ecc35bd69967402b9eeb700614baf46b, State: Running, Role: LEADER
I20260925 01:27:46.969446 25245 consensus_queue.cc:237] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } }
I20260925 01:27:46.973860 24772 catalog_manager.cc:5495] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc35bd69967402b9eeb700614baf46b (127.6.236.131). New cstate: current_term: 1 leader_uuid: "ecc35bd69967402b9eeb700614baf46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc35bd69967402b9eeb700614baf46b" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 35983 } health_report { overall_health: HEALTHY } } }
I20260925 01:27:47.007623 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:47.002516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.6.236.131:35983)
0925 01:27:47.002725 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:47.002730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:47.002803 (+ 73us) server_negotiation.cc:407] Connection header received
0925 01:27:47.002987 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:47.002991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:47.003075 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:47.003185 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:47.003505 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.004267 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.005615 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.006019 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.006111 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:47.006611 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:47.007132 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:47.007248 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:47.007311 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 01:27:47.007373 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:27:47.015712 24915 raft_consensus.cc:1261] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 1 FOLLOWER]: Refusing update from remote peer ecc35bd69967402b9eeb700614baf46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:47.015712 25046 raft_consensus.cc:1261] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 1 FOLLOWER]: Refusing update from remote peer ecc35bd69967402b9eeb700614baf46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:47.016755 25245 consensus_queue.cc:1035] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a79f5d6a8d044c0c8237438a4b6897cf" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 43521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:47.017086 25249 consensus_queue.cc:1035] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e58d13f76a0470c83c904234b5a0037" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:47.026942 25252 mvcc.cc:205] Tried to move back new op lower bound from 7333067436085784576 to 7333067435910561792. Current Snapshot: MvccSnapshot[applied={T|T < 7333067436085784576}]
I20260925 01:27:47.027314 25253 mvcc.cc:205] Tried to move back new op lower bound from 7333067436085784576 to 7333067435910561792. Current Snapshot: MvccSnapshot[applied={T|T < 7333067436085784576}]
I20260925 01:27:47.032269 25254 mvcc.cc:205] Tried to move back new op lower bound from 7333067436085784576 to 7333067435910561792. Current Snapshot: MvccSnapshot[applied={T|T < 7333067436085784576}]
I20260925 01:27:47.041590 24772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:47.041878 24772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:47.045460 24772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bc40af734074a91ab8135f1cb18839f: Sending DeleteTablet for 3 replicas of tablet 04fd776f75734fceba5011b38c56d8cc
I20260925 01:27:47.046597 24895 tablet_service.cc:1514] Processing DeleteTablet for tablet 04fd776f75734fceba5011b38c56d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:47 UTC) from {username='slave'} at 127.0.0.1:37438
I20260925 01:27:47.046587 25026 tablet_service.cc:1514] Processing DeleteTablet for tablet 04fd776f75734fceba5011b38c56d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:47 UTC) from {username='slave'} at 127.0.0.1:34766
I20260925 01:27:47.046669 25156 tablet_service.cc:1514] Processing DeleteTablet for tablet 04fd776f75734fceba5011b38c56d8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:47 UTC) from {username='slave'} at 127.0.0.1:55176
I20260925 01:27:47.051833 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24833
I20260925 01:27:47.059665 25262 tablet_replica.cc:330] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: stopping tablet replica
I20260925 01:27:47.060278 25264 tablet_replica.cc:330] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: stopping tablet replica
I20260925 01:27:47.061105 25264 raft_consensus.cc:2229] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:47.061672 25262 raft_consensus.cc:2229] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:47.062004 25263 tablet_replica.cc:330] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: stopping tablet replica
I20260925 01:27:47.062407 25263 raft_consensus.cc:2229] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:47.062769 25262 raft_consensus.cc:2258] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:47.062829 25263 raft_consensus.cc:2258] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:47.063114 25264 raft_consensus.cc:2258] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:47.065275 25263 ts_tablet_manager.cc:1905] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:47.065275 25262 ts_tablet_manager.cc:1905] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:47.068645 25264 ts_tablet_manager.cc:1905] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:47.072257 25262 ts_tablet_manager.cc:1918] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:47.072266 25263 ts_tablet_manager.cc:1918] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:47.072410 25263 log.cc:1196] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/04fd776f75734fceba5011b38c56d8cc
I20260925 01:27:47.072538 25262 log.cc:1196] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/04fd776f75734fceba5011b38c56d8cc
I20260925 01:27:47.072973 25263 ts_tablet_manager.cc:1939] T 04fd776f75734fceba5011b38c56d8cc P 5e58d13f76a0470c83c904234b5a0037: Deleting consensus metadata
I20260925 01:27:47.073125 25262 ts_tablet_manager.cc:1939] T 04fd776f75734fceba5011b38c56d8cc P a79f5d6a8d044c0c8237438a4b6897cf: Deleting consensus metadata
I20260925 01:27:47.074810 24759 catalog_manager.cc:4841] TS 5e58d13f76a0470c83c904234b5a0037 (127.6.236.130:42861): tablet 04fd776f75734fceba5011b38c56d8cc (table test-table [id=36c164e412e84cf1b0de85f1323cd5d5]) successfully deleted
I20260925 01:27:47.075121 24760 catalog_manager.cc:4841] TS a79f5d6a8d044c0c8237438a4b6897cf (127.6.236.129:43521): tablet 04fd776f75734fceba5011b38c56d8cc (table test-table [id=36c164e412e84cf1b0de85f1323cd5d5]) successfully deleted
I20260925 01:27:47.075747 25264 ts_tablet_manager.cc:1918] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:47.075917 25264 log.cc:1196] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/04fd776f75734fceba5011b38c56d8cc
I20260925 01:27:47.076419 25264 ts_tablet_manager.cc:1939] T 04fd776f75734fceba5011b38c56d8cc P ecc35bd69967402b9eeb700614baf46b: Deleting consensus metadata
I20260925 01:27:47.078014 24759 catalog_manager.cc:4841] TS ecc35bd69967402b9eeb700614baf46b (127.6.236.131:35983): tablet 04fd776f75734fceba5011b38c56d8cc (table test-table [id=36c164e412e84cf1b0de85f1323cd5d5]) successfully deleted
I20260925 01:27:47.078513 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:47.053108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.6.236.129:43521)
0925 01:27:47.053279 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:47.053285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:47.055267 (+ 1982us) server_negotiation.cc:407] Connection header received
0925 01:27:47.055451 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:47.055457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:47.055555 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:47.055662 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:47.060558 (+ 4896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.061759 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.063725 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.064410 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.064504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:47.066418 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:47.066430 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:47.066450 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:47.067217 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:47.067277 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:47.067315 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:47.067453 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:47.078015 (+ 10562us) server_negotiation.cc:299] Negotiation successful
0925 01:27:47.078146 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:27:47.079671 24955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:47.166406 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24833
I20260925 01:27:47.188516 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24964
I20260925 01:27:47.197381 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:47.189549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.6.236.130:42861)
0925 01:27:47.189992 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:47.189998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:47.190018 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:27:47.190083 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:47.190087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:47.190215 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:47.190340 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:47.190999 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.191844 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.193721 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.194519 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.194652 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:47.195881 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:47.195893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:47.195912 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:47.196659 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:47.196732 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:47.196760 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:47.196878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:47.196924 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 01:27:47.197028 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260925 01:27:47.198079 25086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:47.291345 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24964
I20260925 01:27:47.312052 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25094
I20260925 01:27:47.324254 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:47.312854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.6.236.131:35983)
0925 01:27:47.313060 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:47.313066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:47.313088 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:27:47.313207 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:47.313212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:47.313314 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:47.313441 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:47.313797 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.315057 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.316275 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:47.316940 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:47.317063 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:47.322824 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:47.322833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:47.322848 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:47.323462 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:47.323511 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:47.323532 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:47.323616 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:47.323810 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 01:27:47.323908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:27:47.324992 25216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:47.443558 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25094
I20260925 01:27:47.459790 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 24743
I20260925 01:27:47.460852 24804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:47.567057 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 24743
2026-09-25T01:27:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:27:47 dist-test-slave-s0b2 krb5kdc[25272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:27:47 dist-test-slave-s0b2 krb5kdc[25272](info): set up 2 sockets
Sep 25 01:27:47 dist-test-slave-s0b2 krb5kdc[25272](info): commencing operation
krb5kdc: starting...
W20260925 01:27:48.578948 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:27:48 dist-test-slave-s0b2 krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:27:48.803375 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35483
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:35483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:49.118733 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:49.119017 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:49.119146 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:49.126206 25285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:49.126319 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:49.126391 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:49.126467 25285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:49.126525 25285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:49.135898 25285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:35483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:49.137269 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:49.138602 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:49.149595 25290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:49.149550 25293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:49.149545 25291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:49.150415 25285 server_base.cc:1034] running on GCE node
I20260925 01:27:49.151155 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:49.152499 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:49.153713 25285 hybrid_clock.cc:648] HybridClock initialized: now 1790299669153653 us; error 43 us; skew 500 ppm
I20260925 01:27:49.154037 25285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:49 dist-test-slave-s0b2 krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299669, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:49.156988 25285 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:27:49.158469 25285 webserver.cc:460] Webserver started at http://127.6.236.190:40077/ using document root <none> and password file <none>
I20260925 01:27:49.159147 25285 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:49.159262 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:49.159549 25285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:49.162640 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "d8e02b5c822441f5ad37fe43a37ad86e"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "be0f595724502e450e3a7e033bd847a4"
server_key_iv: "f59a65a71dbf7d31aa704c37f3a1df87"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.163358 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "d8e02b5c822441f5ad37fe43a37ad86e"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "be0f595724502e450e3a7e033bd847a4"
server_key_iv: "f59a65a71dbf7d31aa704c37f3a1df87"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.168360 25285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:27:49.171942 25300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:49.173343 25285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:27:49.173497 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "d8e02b5c822441f5ad37fe43a37ad86e"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "be0f595724502e450e3a7e033bd847a4"
server_key_iv: "f59a65a71dbf7d31aa704c37f3a1df87"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.173681 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:49.204632 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:49.208474 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:49.208737 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:49.225978 25285 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:35483
I20260925 01:27:49.225989 25351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:35483 every 8 connection(s)
I20260925 01:27:49.227447 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:49.231129 25352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:49.233172 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25285
I20260925 01:27:49.233366 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:49.233906 7090 external_mini_cluster.cc:1428] Setting key 9425737d0e7a046f2410542911f26d8e
I20260925 01:27:49.243474 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: Bootstrap starting.
I20260925 01:27:49.247269 25352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: Neither blocks nor log segments found. Creating new log.
Sep 25 01:27:49 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:49.248445 25352 log.cc:824] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:49.251925 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: No bootstrap required, opened a new log
I20260925 01:27:49.257826 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:49.235799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.6.236.190:35483)
0925 01:27:49.236598 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:49.236613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:49.236653 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:27:49.238391 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:49.238416 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:49.238809 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:49.239123 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:49.240970 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.242336 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.244079 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.244724 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.244854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:49.249625 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:49.249632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:49.249647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:49.249690 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:49.253403 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:49.254043 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:49.254051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:49.254060 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:49.254308 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:49.254637 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:49.254643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:49.254647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:49.255370 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:49.255664 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:49.255997 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:27:49.256439 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":351,"threads_started":1}
I20260925 01:27:49.261101 25352 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } }
I20260925 01:27:49.261440 25352 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:49.261518 25352 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e02b5c822441f5ad37fe43a37ad86e, State: Initialized, Role: FOLLOWER
I20260925 01:27:49.261992 25352 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } }
I20260925 01:27:49.262189 25352 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:49.262328 25352 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:49.262475 25352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:49.264505 25352 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } }
I20260925 01:27:49.264976 25352 leader_election.cc:302] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e02b5c822441f5ad37fe43a37ad86e; no voters:
I20260925 01:27:49.265506 25352 leader_election.cc:288] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:49.265616 25357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:49.266215 25357 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [term 1 LEADER]: Becoming Leader. State: Replica: d8e02b5c822441f5ad37fe43a37ad86e, State: Running, Role: LEADER
I20260925 01:27:49.266731 25357 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } }
I20260925 01:27:49.267448 25352 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:49.269133 25358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } } }
I20260925 01:27:49.269454 25358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:49.269846 25359 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e02b5c822441f5ad37fe43a37ad86e. Latest consensus state: current_term: 1 leader_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e02b5c822441f5ad37fe43a37ad86e" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35483 } } }
I20260925 01:27:49.270134 25359 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:49.271036 25363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:49.276677 25363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:49.285856 25363 catalog_manager.cc:1347] Generated new cluster ID: ff26d7fdc0134f9c875a8948ec9d8423
I20260925 01:27:49.285959 25363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:49.316303 25363 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:49.317600 25363 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:49.324496 25363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: Generated new TSK 0
I20260925 01:27:49.325459 25363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:27:49.393532 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:49.712082 25378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:49.712370 25378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:49.712503 25378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:49.719810 25378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:49.719921 25378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:49.720072 25378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:49.729876 25378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:49.731247 25378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:49.732568 25378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:49.744904 25383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:49.745316 25386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:49.745347 25384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:49.745803 25378 server_base.cc:1034] running on GCE node
I20260925 01:27:49.746376 25378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:49.747299 25378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:49.748499 25378 hybrid_clock.cc:648] HybridClock initialized: now 1790299669748444 us; error 46 us; skew 500 ppm
I20260925 01:27:49.748773 25378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:49 dist-test-slave-s0b2 krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299669, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:49.751837 25378 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:27:49.753224 25378 webserver.cc:460] Webserver started at http://127.6.236.129:41285/ using document root <none> and password file <none>
I20260925 01:27:49.753875 25378 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:49.753990 25378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:49.754331 25378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:49.757357 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "d01e0dc63c9a4dd8b9afd6a000fed422"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "2ab9e431d589d288ac4c8699ea1dda8d"
server_key_iv: "417557e5829d88ed00804adeab46bdec"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.758175 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "d01e0dc63c9a4dd8b9afd6a000fed422"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "2ab9e431d589d288ac4c8699ea1dda8d"
server_key_iv: "417557e5829d88ed00804adeab46bdec"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.763500 25378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:27:49.767546 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:49.769014 25378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:49.769186 25378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "d01e0dc63c9a4dd8b9afd6a000fed422"
format_stamp: "Formatted at 2026-09-25 01:27:49 on dist-test-slave-s0b2"
server_key: "2ab9e431d589d288ac4c8699ea1dda8d"
server_key_iv: "417557e5829d88ed00804adeab46bdec"
server_key_version: "encryptionkey@0"
I20260925 01:27:49.769379 25378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:49.783752 25378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:49.787741 25378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:49.787994 25378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:49.788893 25378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:49.790126 25378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:49.790202 25378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:49.790342 25378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:49.790393 25378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:49.818006 25378 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:42513
I20260925 01:27:49.818042 25505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:42513 every 8 connection(s)
I20260925 01:27:49.819285 25378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:49.828668 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25378
I20260925 01:27:49.828847 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:49.829391 7090 external_mini_cluster.cc:1428] Setting key 0093ce1bffa3f8a28666acb3c037f0a7
Sep 25 01:27:49 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299669, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:49.839207 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:49.821753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:42919 (local address 127.6.236.190:35483)
0925 01:27:49.822209 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:49.822215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:49.823137 (+ 922us) server_negotiation.cc:407] Connection header received
0925 01:27:49.824990 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:49.824994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:49.825065 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:49.825158 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:49.826635 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.827466 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.829267 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.829695 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.829795 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:49.833535 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:49.833540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:49.833544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:49.833576 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:49.835947 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:49.836738 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:49.836742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:49.836745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:49.836856 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:49.837325 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:49.837330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:49.837334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:49.837750 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:49.837921 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:49.838781 (+ 860us) server_negotiation.cc:299] Negotiation successful
0925 01:27:49.838953 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":206,"threads_started":1}
I20260925 01:27:49.840315 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:49.822279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35483 (local address 127.6.236.129:42919)
0925 01:27:49.822968 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:49.823009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:49.824754 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:49.825393 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:49.825401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:49.825891 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:49.826499 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:49.826519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.827596 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.827601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:49.829134 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:49.829150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:49.829920 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:49.829927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:49.830128 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:49.830773 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:49.830803 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:49.833324 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:49.836078 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:49.836094 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:49.836112 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:49.836629 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:49.836938 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:49.836943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:49.836946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:49.837172 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:49.837886 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:49.837921 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:49.838401 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 01:27:49.838663 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 01:27:49.839157 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":262,"threads_started":1}
I20260925 01:27:49.841928 25506 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35483
I20260925 01:27:49.842301 25506 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:49.843127 25506 heartbeater.cc:511] Master 127.6.236.190:35483 requested a full tablet report, sending...
I20260925 01:27:49.845258 25317 ts_manager.cc:194] Registered new tserver with Master: d01e0dc63c9a4dd8b9afd6a000fed422 (127.6.236.129:42513)
I20260925 01:27:49.846743 25317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:42919
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:27:49.884720 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:50.196170 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:50.196444 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:50.196585 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:50.203958 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:50.204072 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:50.204222 25512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:50.213997 25512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:50.215220 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:50.216498 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:50.228801 25517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:50.228775 25518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:50.228937 25520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:50.230044 25512 server_base.cc:1034] running on GCE node
I20260925 01:27:50.230654 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:50.231613 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:50.232834 25512 hybrid_clock.cc:648] HybridClock initialized: now 1790299670232787 us; error 43 us; skew 500 ppm
I20260925 01:27:50.233114 25512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:50 dist-test-slave-s0b2 krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:50.236572 25512 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:27:50.238005 25512 webserver.cc:460] Webserver started at http://127.6.236.130:40377/ using document root <none> and password file <none>
I20260925 01:27:50.238691 25512 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:50.238801 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:50.239095 25512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:50.241861 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "f36fc72fbb1942689cbab63470f366b3"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "0fe9ad10d46bc92b6e71291d7eab240d"
server_key_iv: "30417ca99a7a9d43a6d7c5ba116be3f9"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.242612 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "f36fc72fbb1942689cbab63470f366b3"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "0fe9ad10d46bc92b6e71291d7eab240d"
server_key_iv: "30417ca99a7a9d43a6d7c5ba116be3f9"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.247605 25512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:27:50.251040 25527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.252455 25512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:50.252620 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "f36fc72fbb1942689cbab63470f366b3"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "0fe9ad10d46bc92b6e71291d7eab240d"
server_key_iv: "30417ca99a7a9d43a6d7c5ba116be3f9"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.252801 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:50.269380 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:50.273211 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:50.273460 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:50.274474 25512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:50.275789 25512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:50.275856 25512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.275969 25512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:50.276015 25512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.304821 25512 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:36743
I20260925 01:27:50.304862 25639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:36743 every 8 connection(s)
I20260925 01:27:50.306032 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:50.315563 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25512
I20260925 01:27:50.315737 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:27:50.316165 7090 external_mini_cluster.cc:1428] Setting key 25c3873afe41e301445b033754810e27
Sep 25 01:27:50 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:50.324926 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.308543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:57397 (local address 127.6.236.190:35483)
0925 01:27:50.308741 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.308746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.309833 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 01:27:50.311656 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.311662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.311752 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.311837 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:50.313303 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.314037 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.315551 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.316039 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.316145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.319630 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:50.319635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:50.319638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:50.319670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:50.322066 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.322760 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.322764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.322767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.322889 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.323288 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.323292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.323294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.323543 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:50.323673 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.324340 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.324561 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:27:50.325984 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.308979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35483 (local address 127.6.236.130:57397)
0925 01:27:50.309672 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:50.309716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:50.311395 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:50.312080 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:50.312088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:50.312594 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:50.313134 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.313151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.314179 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.314183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.315419 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.315431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.316265 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.316273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.316478 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.317096 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:50.317117 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:50.319426 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:50.322188 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:50.322197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:50.322212 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:50.322640 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:50.322989 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:50.322994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:50.322999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:50.323178 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:50.323659 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:50.323667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:50.323991 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 01:27:50.324217 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 01:27:50.324719 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
I20260925 01:27:50.327335 25640 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35483
I20260925 01:27:50.327615 25640 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:50.328310 25640 heartbeater.cc:511] Master 127.6.236.190:35483 requested a full tablet report, sending...
I20260925 01:27:50.329700 25317 ts_manager.cc:194] Registered new tserver with Master: f36fc72fbb1942689cbab63470f366b3 (127.6.236.130:36743)
I20260925 01:27:50.330509 25317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:57397
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:27:50.366869 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:34441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:50.676898 25645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:50.677163 25645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:50.677284 25645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:50.684312 25645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:50.684412 25645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:50.684520 25645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:50.693822 25645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:34441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:50.695019 25645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:50.696269 25645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:50.708523 25650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:50.708525 25653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:50.708590 25651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:50.709240 25645 server_base.cc:1034] running on GCE node
I20260925 01:27:50.709805 25645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:50.710753 25645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:50.711943 25645 hybrid_clock.cc:648] HybridClock initialized: now 1790299670711890 us; error 46 us; skew 500 ppm
I20260925 01:27:50.712226 25645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:50 dist-test-slave-s0b2 krb5kdc[25272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:50.715173 25645 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:27:50.716454 25645 webserver.cc:460] Webserver started at http://127.6.236.131:39123/ using document root <none> and password file <none>
I20260925 01:27:50.717090 25645 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:50.717185 25645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:50.717429 25645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:50.720178 25645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "146756e5d3914770b4cb7eb876d3388c"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "3c52e1d9b797e260b394bfb9ae0469f4"
server_key_iv: "e78d3dcf02fe350a77e542430c539d66"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.720867 25645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "146756e5d3914770b4cb7eb876d3388c"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "3c52e1d9b797e260b394bfb9ae0469f4"
server_key_iv: "e78d3dcf02fe350a77e542430c539d66"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.725571 25645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 01:27:50.728768 25660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.729984 25645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 01:27:50.730105 25645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "146756e5d3914770b4cb7eb876d3388c"
format_stamp: "Formatted at 2026-09-25 01:27:50 on dist-test-slave-s0b2"
server_key: "3c52e1d9b797e260b394bfb9ae0469f4"
server_key_iv: "e78d3dcf02fe350a77e542430c539d66"
server_key_version: "encryptionkey@0"
I20260925 01:27:50.730314 25645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:50.760286 25645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:50.764313 25645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:50.764535 25645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:50.765372 25645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:50.766817 25645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:50.766906 25645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.767023 25645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:50.767089 25645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:50.796200 25645 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44109
I20260925 01:27:50.796241 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44109 every 8 connection(s)
I20260925 01:27:50.797429 25645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:50.798484 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25645
I20260925 01:27:50.798660 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:27:50.799225 7090 external_mini_cluster.cc:1428] Setting key 1678cbf39dbdc84a99be9593842e43de
Sep 25 01:27:50 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:50.816854 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.800463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:43435 (local address 127.6.236.190:35483)
0925 01:27:50.800794 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.800801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.801970 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 01:27:50.804028 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.804032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.804130 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.804218 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:50.805631 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.806410 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.807948 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.808336 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.808424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.811664 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:50.811669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:50.811672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:50.811716 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:50.814299 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.814945 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.814949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.814951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.815082 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.815438 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.815441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.815443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.815722 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:50.815834 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.816435 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.816607 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 01:27:50.817766 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.800981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35483 (local address 127.6.236.131:43435)
0925 01:27:50.801823 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:50.801875 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:50.803768 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:50.804401 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:50.804409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:50.804945 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:50.805488 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.805506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.806595 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.806599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.807836 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.807848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.808519 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.808524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.808755 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.809320 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:50.809342 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:50.811456 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:50.814424 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:50.814430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:50.814442 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:50.814842 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:50.815161 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:50.815164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:50.815166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:50.815339 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:50.815817 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:50.815823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:50.816117 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 01:27:50.816332 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 01:27:50.816690 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":342,"threads_started":1}
I20260925 01:27:50.819094 25773 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35483
I20260925 01:27:50.819411 25773 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:50.820248 25773 heartbeater.cc:511] Master 127.6.236.190:35483 requested a full tablet report, sending...
I20260925 01:27:50.821805 25317 ts_manager.cc:194] Registered new tserver with Master: 146756e5d3914770b4cb7eb876d3388c (127.6.236.131:44109)
I20260925 01:27:50.822636 25317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:43435
I20260925 01:27:50.826253 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:50.841943 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.829924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.6.236.190:35483)
0925 01:27:50.830130 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.830137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.830472 (+ 335us) server_negotiation.cc:407] Connection header received
0925 01:27:50.830598 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.830601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.830671 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.830749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:50.831974 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.832664 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.833968 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.834453 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.834544 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.836866 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:50.836871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:50.836874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:50.836903 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:50.839340 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.840055 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.840059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.840061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.840186 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:50.840591 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:50.840595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:50.840597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:50.840884 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:50.841125 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.841488 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.841675 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:27:50.849969 25506 heartbeater.cc:503] Master 127.6.236.190:35483 was elected leader, sending a full tablet report...
I20260925 01:27:50.845413 25317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:50.852203 25317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:50.884191 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.863652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.6.236.129:42513)
0925 01:27:50.876388 (+ 12736us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.876397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.876429 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:27:50.876529 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.876538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.876796 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.877023 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:50.877872 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.879335 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.882002 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.883036 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.883160 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.883398 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.883714 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.883878 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12564,"thread_start_us":12407,"threads_started":1}
I20260925 01:27:50.884284 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.863480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:42513 (local address 127.0.0.1:39812)
0925 01:27:50.875570 (+ 12090us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:50.875620 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:50.875814 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:50.877007 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:50.877014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:50.877044 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:50.877615 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.877630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.879460 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.879465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.881836 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.881851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.883236 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.883244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.883495 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.883534 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:27:50.883642 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:27:50.883828 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11818,"mutex_wait_us":25,"thread_start_us":13342,"threads_started":1}
I20260925 01:27:50.885465 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.863511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36743 (local address 127.0.0.1:44152)
0925 01:27:50.877120 (+ 13609us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:50.877138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:50.877274 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:50.878472 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:50.878477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:50.878500 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:50.878857 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.878869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.880575 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.880581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.882635 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.882650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.884776 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.884783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.884933 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.884963 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:27:50.885052 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:27:50.885166 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13465,"thread_start_us":11674,"threads_started":1}
I20260925 01:27:50.885586 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.863736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.6.236.130:36743)
0925 01:27:50.876669 (+ 12933us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.876677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.877919 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 01:27:50.878034 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.878042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.878289 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.878459 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:50.879011 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.880410 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.882856 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.884246 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.884697 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.884897 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.885109 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.885266 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12780,"thread_start_us":12674,"threads_started":1}
I20260925 01:27:50.887744 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.877415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44109 (local address 127.0.0.1:42290)
0925 01:27:50.877986 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:50.878002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:50.878098 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:50.880361 (+ 2263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:50.880365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:50.880387 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:50.880872 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.880884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.882192 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.882197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.884671 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:50.884686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.886979 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.886986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:50.887147 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.887178 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:27:50.887289 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:27:50.887415 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":58,"thread_start_us":306,"threads_started":1}
I20260925 01:27:50.887805 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:50.877533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.6.236.131:44109)
0925 01:27:50.879600 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:50.879606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:50.879627 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:27:50.879711 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:50.879717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:50.879927 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:50.880104 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:50.881033 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.882048 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.884850 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:50.886512 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:50.886635 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:50.886846 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:50.887352 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 01:27:50.887501 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1945,"thread_start_us":1249,"threads_started":1}
I20260925 01:27:50.888270 25441 tablet_service.cc:1467] Processing CreateTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f (DEFAULT_TABLE table=test-table [id=84fabb94eaa8431cbe50545ca0588a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:50.889055 25575 tablet_service.cc:1467] Processing CreateTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f (DEFAULT_TABLE table=test-table [id=84fabb94eaa8431cbe50545ca0588a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:50.889645 25441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7882a0627e074416b2e2d1c3ec36a13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:50.890151 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7882a0627e074416b2e2d1c3ec36a13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:50.891356 25708 tablet_service.cc:1467] Processing CreateTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f (DEFAULT_TABLE table=test-table [id=84fabb94eaa8431cbe50545ca0588a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:50.892693 25708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7882a0627e074416b2e2d1c3ec36a13f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:50.900640 25791 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Bootstrap starting.
I20260925 01:27:50.903631 25792 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Bootstrap starting.
I20260925 01:27:50.904919 25791 tablet_bootstrap.cc:654] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:50.906183 25791 log.cc:824] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:50.906978 25793 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Bootstrap starting.
I20260925 01:27:50.907097 25792 tablet_bootstrap.cc:654] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:50.908370 25792 log.cc:824] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:50.909933 25793 tablet_bootstrap.cc:654] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:50.911180 25792 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: No bootstrap required, opened a new log
I20260925 01:27:50.911139 25793 log.cc:824] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:50.911443 25792 ts_tablet_manager.cc:1397] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 01:27:50.911572 25791 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: No bootstrap required, opened a new log
I20260925 01:27:50.911852 25791 ts_tablet_manager.cc:1397] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 01:27:50.914074 25793 tablet_bootstrap.cc:492] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: No bootstrap required, opened a new log
I20260925 01:27:50.914367 25793 ts_tablet_manager.cc:1397] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 01:27:50.919059 25791 raft_consensus.cc:348] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.919425 25791 raft_consensus.cc:374] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:50.919517 25791 raft_consensus.cc:729] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36fc72fbb1942689cbab63470f366b3, State: Initialized, Role: FOLLOWER
I20260925 01:27:50.920065 25791 consensus_queue.cc:260] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.921126 25640 heartbeater.cc:503] Master 127.6.236.190:35483 was elected leader, sending a full tablet report...
I20260925 01:27:50.921811 25791 ts_tablet_manager.cc:1428] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 01:27:50.921639 25792 raft_consensus.cc:348] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.922022 25792 raft_consensus.cc:374] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:50.922103 25792 raft_consensus.cc:729] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01e0dc63c9a4dd8b9afd6a000fed422, State: Initialized, Role: FOLLOWER
I20260925 01:27:50.922751 25792 consensus_queue.cc:260] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.924431 25792 ts_tablet_manager.cc:1428] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 01:27:50.924371 25793 raft_consensus.cc:348] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.924783 25793 raft_consensus.cc:374] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:50.924887 25793 raft_consensus.cc:729] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146756e5d3914770b4cb7eb876d3388c, State: Initialized, Role: FOLLOWER
I20260925 01:27:50.925463 25793 consensus_queue.cc:260] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:50.926550 25773 heartbeater.cc:503] Master 127.6.236.190:35483 was elected leader, sending a full tablet report...
I20260925 01:27:50.927784 25793 ts_tablet_manager.cc:1428] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260925 01:27:51.048974 25774 tablet.cc:2369] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:51.057824 25641 tablet.cc:2369] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:51.071453 25507 tablet.cc:2369] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:51.106916 25798 raft_consensus.cc:482] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:51.107221 25798 raft_consensus.cc:504] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:51.108899 25798 leader_election.cc:288] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f36fc72fbb1942689cbab63470f366b3 (127.6.236.130:36743), 146756e5d3914770b4cb7eb876d3388c (127.6.236.131:44109)
I20260925 01:27:51.115507 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.109477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:59059 (local address 127.6.236.131:44109)
0925 01:27:51.109714 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.109722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.110508 (+ 786us) server_negotiation.cc:407] Connection header received
0925 01:27:51.110658 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.110662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.110731 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.110818 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:51.111422 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.112120 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.113825 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.114614 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.114753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.114826 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.115146 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.115271 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:27:51.115509 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.109427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:41519 (local address 127.6.236.130:36743)
0925 01:27:51.109725 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.109732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.110371 (+ 639us) server_negotiation.cc:407] Connection header received
0925 01:27:51.110548 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.110554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.110631 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.110706 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:51.111271 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.112006 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.113855 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.114646 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.114811 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.114880 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.115159 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.115277 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 01:27:51.115674 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.109155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44109 (local address 127.6.236.129:59059)
0925 01:27:51.110400 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:51.110417 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:51.110548 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:51.110850 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:51.110854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:51.110890 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:51.111272 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:51.111284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.112253 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.112257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:51.113676 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:51.113692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.114820 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.114827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:51.114992 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.115026 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:27:51.115103 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:27:51.115237 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":24,"thread_start_us":765,"threads_started":1}
I20260925 01:27:51.116009 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.109122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:36743 (local address 127.6.236.129:41519)
0925 01:27:51.110262 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:51.110283 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:51.110414 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:51.110742 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:51.110747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:51.110771 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:51.111125 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:51.111137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.112129 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.112133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:51.113704 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:51.113716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.114865 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.114871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:51.115013 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.115039 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:27:51.115120 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:27:51.115242 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":53,"thread_start_us":832,"threads_started":1}
I20260925 01:27:51.116645 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7882a0627e074416b2e2d1c3ec36a13f" candidate_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146756e5d3914770b4cb7eb876d3388c" is_pre_election: true
I20260925 01:27:51.116910 25595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7882a0627e074416b2e2d1c3ec36a13f" candidate_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36fc72fbb1942689cbab63470f366b3" is_pre_election: true
I20260925 01:27:51.117233 25728 raft_consensus.cc:2454] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01e0dc63c9a4dd8b9afd6a000fed422 in term 0.
I20260925 01:27:51.117442 25595 raft_consensus.cc:2454] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01e0dc63c9a4dd8b9afd6a000fed422 in term 0.
I20260925 01:27:51.117937 25397 leader_election.cc:302] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146756e5d3914770b4cb7eb876d3388c, d01e0dc63c9a4dd8b9afd6a000fed422; no voters:
I20260925 01:27:51.118378 25798 raft_consensus.cc:2781] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:51.118512 25798 raft_consensus.cc:482] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:51.118613 25798 raft_consensus.cc:3037] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:51.120685 25798 raft_consensus.cc:504] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:51.121243 25798 leader_election.cc:288] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [CANDIDATE]: Term 1 election: Requested vote from peers f36fc72fbb1942689cbab63470f366b3 (127.6.236.130:36743), 146756e5d3914770b4cb7eb876d3388c (127.6.236.131:44109)
I20260925 01:27:51.121665 25595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7882a0627e074416b2e2d1c3ec36a13f" candidate_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36fc72fbb1942689cbab63470f366b3"
I20260925 01:27:51.121694 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7882a0627e074416b2e2d1c3ec36a13f" candidate_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146756e5d3914770b4cb7eb876d3388c"
I20260925 01:27:51.121827 25595 raft_consensus.cc:3037] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:51.121886 25728 raft_consensus.cc:3037] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:51.123899 25595 raft_consensus.cc:2454] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01e0dc63c9a4dd8b9afd6a000fed422 in term 1.
I20260925 01:27:51.124002 25728 raft_consensus.cc:2454] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01e0dc63c9a4dd8b9afd6a000fed422 in term 1.
I20260925 01:27:51.124253 25395 leader_election.cc:302] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d01e0dc63c9a4dd8b9afd6a000fed422, f36fc72fbb1942689cbab63470f366b3; no voters:
I20260925 01:27:51.124550 25798 raft_consensus.cc:2781] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:51.124953 25798 raft_consensus.cc:686] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 1 LEADER]: Becoming Leader. State: Replica: d01e0dc63c9a4dd8b9afd6a000fed422, State: Running, Role: LEADER
I20260925 01:27:51.125425 25798 consensus_queue.cc:237] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } }
I20260925 01:27:51.129746 25316 catalog_manager.cc:5495] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 reported cstate change: term changed from 0 to 1, leader changed from <none> to d01e0dc63c9a4dd8b9afd6a000fed422 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d01e0dc63c9a4dd8b9afd6a000fed422" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 } health_report { overall_health: UNKNOWN } } }
I20260925 01:27:51.169314 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.163899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.6.236.129:42513)
0925 01:27:51.164132 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.164137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.164155 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:27:51.164239 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.164242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.164331 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.164464 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:51.164852 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.165555 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.167043 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.167471 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.167579 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.168205 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:51.168786 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:51.168923 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.169052 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.169098 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:27:51.177757 25728 raft_consensus.cc:1261] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 1 FOLLOWER]: Refusing update from remote peer d01e0dc63c9a4dd8b9afd6a000fed422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:51.177750 25595 raft_consensus.cc:1261] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 1 FOLLOWER]: Refusing update from remote peer d01e0dc63c9a4dd8b9afd6a000fed422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:51.178828 25802 consensus_queue.cc:1035] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f36fc72fbb1942689cbab63470f366b3" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 36743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:51.179169 25798 consensus_queue.cc:1035] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146756e5d3914770b4cb7eb876d3388c" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:51.214931 25316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:51.215238 25316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:51.218858 25316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8e02b5c822441f5ad37fe43a37ad86e: Sending DeleteTablet for 3 replicas of tablet 7882a0627e074416b2e2d1c3ec36a13f
I20260925 01:27:51.219949 25441 tablet_service.cc:1514] Processing DeleteTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:51 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:39812
I20260925 01:27:51.220088 25575 tablet_service.cc:1514] Processing DeleteTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:51 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:44152
I20260925 01:27:51.220824 25815 tablet_replica.cc:330] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: stopping tablet replica
I20260925 01:27:51.220777 25708 tablet_service.cc:1514] Processing DeleteTablet for tablet 7882a0627e074416b2e2d1c3ec36a13f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:51 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42290
I20260925 01:27:51.221338 25816 tablet_replica.cc:330] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: stopping tablet replica
I20260925 01:27:51.221324 25815 raft_consensus.cc:2229] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:51.221784 25816 raft_consensus.cc:2229] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:51.222098 25815 raft_consensus.cc:2258] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:51.222297 25816 raft_consensus.cc:2258] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:51.224251 25816 ts_tablet_manager.cc:1905] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:51.224283 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25378
I20260925 01:27:51.230983 25816 ts_tablet_manager.cc:1918] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:51.231246 25816 log.cc:1196] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/7882a0627e074416b2e2d1c3ec36a13f
I20260925 01:27:51.231846 25816 ts_tablet_manager.cc:1939] T 7882a0627e074416b2e2d1c3ec36a13f P f36fc72fbb1942689cbab63470f366b3: Deleting consensus metadata
I20260925 01:27:51.233875 25817 tablet_replica.cc:330] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: stopping tablet replica
I20260925 01:27:51.233891 25302 catalog_manager.cc:4841] TS f36fc72fbb1942689cbab63470f366b3 (127.6.236.130:36743): tablet 7882a0627e074416b2e2d1c3ec36a13f (table test-table [id=84fabb94eaa8431cbe50545ca0588a62]) successfully deleted
I20260925 01:27:51.234443 25817 raft_consensus.cc:2229] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:51.234926 25817 raft_consensus.cc:2258] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:51.236440 25815 ts_tablet_manager.cc:1905] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:51.237062 25817 ts_tablet_manager.cc:1905] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:51.243711 25817 ts_tablet_manager.cc:1918] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:51.243932 25817 log.cc:1196] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/7882a0627e074416b2e2d1c3ec36a13f
I20260925 01:27:51.244448 25817 ts_tablet_manager.cc:1939] T 7882a0627e074416b2e2d1c3ec36a13f P 146756e5d3914770b4cb7eb876d3388c: Deleting consensus metadata
I20260925 01:27:51.246145 25304 catalog_manager.cc:4841] TS 146756e5d3914770b4cb7eb876d3388c (127.6.236.131:44109): tablet 7882a0627e074416b2e2d1c3ec36a13f (table test-table [id=84fabb94eaa8431cbe50545ca0588a62]) successfully deleted
I20260925 01:27:51.247660 25815 ts_tablet_manager.cc:1918] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:51.247812 25815 log.cc:1196] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/7882a0627e074416b2e2d1c3ec36a13f
I20260925 01:27:51.248275 25815 ts_tablet_manager.cc:1939] T 7882a0627e074416b2e2d1c3ec36a13f P d01e0dc63c9a4dd8b9afd6a000fed422: Deleting consensus metadata
Sep 25 01:27:51 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:27:51.249814 25304 catalog_manager.cc:4841] TS d01e0dc63c9a4dd8b9afd6a000fed422 (127.6.236.129:42513): tablet 7882a0627e074416b2e2d1c3ec36a13f (table test-table [id=84fabb94eaa8431cbe50545ca0588a62]) successfully deleted
I20260925 01:27:51.257027 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.227125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.6.236.129:42513)
0925 01:27:51.227290 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.227296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.238764 (+ 11468us) server_negotiation.cc:407] Connection header received
0925 01:27:51.239204 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.239210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.239289 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.239363 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:51.241142 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.242353 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.244224 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.244927 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.245031 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.251176 (+ 6145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:51.251187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:51.251201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:51.251241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:51.254045 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.254896 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.254902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.254905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.255052 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.255505 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.255510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.255512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.255905 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:51.256140 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.256507 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.256712 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:27:51.257707 25501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:51.365658 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25378
I20260925 01:27:51.381130 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25512
Sep 25 01:27:51 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:27:51.395252 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.381925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.6.236.130:36743)
0925 01:27:51.382105 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.382109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.382127 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:27:51.382269 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.382280 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.382361 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.382450 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:51.383665 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.384409 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.385676 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.386065 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.386158 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.390031 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:51.390046 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:51.390062 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:51.390097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:51.392813 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.393418 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.393426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.393428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.393548 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.393841 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.393844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.393845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.394222 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:51.394484 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.394772 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.394931 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:27:51.395848 25635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:51.484699 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25512
I20260925 01:27:51.499085 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25645
Sep 25 01:27:51 dist-test-slave-s0b2 krb5kdc[25272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:27:51.513530 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:51.499816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.6.236.131:44109)
0925 01:27:51.499998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:51.500003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:51.500024 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:27:51.500217 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:51.500221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:51.500299 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:51.500393 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:51.501695 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.502481 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.503711 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:51.504097 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:51.504210 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:51.508471 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:51.508487 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:51.508504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:51.508543 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:51.510959 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.511592 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.511600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.511603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.511748 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:51.512100 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:51.512104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:51.512106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:51.512448 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:51.512713 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:51.513020 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 01:27:51.513224 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:27:51.514163 25768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:51.601504 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25645
I20260925 01:27:51.615568 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25285
I20260925 01:27:51.616667 25347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:51.725559 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25285
2026-09-25T01:27:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4161 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T01:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:51Z Disabled control of system clock
I20260925 01:27:51.797667 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38395
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:52.110116 25826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:52.110514 25826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:52.110643 25826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:52.117871 25826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:52.117965 25826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:52.118021 25826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:52.118094 25826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:52.118144 25826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:52.127218 25826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:38395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:52.128450 25826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:52.130199 25826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:52.141088 25832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:52.141283 25831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:52.141132 25834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:52.142328 25826 server_base.cc:1034] running on GCE node
I20260925 01:27:52.143013 25826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:52.144358 25826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:52.145556 25826 hybrid_clock.cc:648] HybridClock initialized: now 1790299672145508 us; error 42 us; skew 500 ppm
I20260925 01:27:52.145882 25826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:52.148036 25826 webserver.cc:460] Webserver started at http://127.6.236.190:35619/ using document root <none> and password file <none>
I20260925 01:27:52.148700 25826 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:52.148793 25826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:52.149098 25826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:52.151996 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1222bcf9756f49e19260b684d449f81f"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "937e943741e7b1a05b230e4c153d2ce2"
server_key_iv: "54b89a4d595267da8093730bee67f6b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.152771 25826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1222bcf9756f49e19260b684d449f81f"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "937e943741e7b1a05b230e4c153d2ce2"
server_key_iv: "54b89a4d595267da8093730bee67f6b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.157817 25826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:27:52.161242 25840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:52.162597 25826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:52.162776 25826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1222bcf9756f49e19260b684d449f81f"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "937e943741e7b1a05b230e4c153d2ce2"
server_key_iv: "54b89a4d595267da8093730bee67f6b3"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.162968 25826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:52.179479 25826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:52.180320 25826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:52.180581 25826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:52.197671 25826 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:38395
I20260925 01:27:52.197690 25891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:38395 every 8 connection(s)
I20260925 01:27:52.199046 25826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:52.202670 25892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:52.208071 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25826
I20260925 01:27:52.208269 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:52.208838 7090 external_mini_cluster.cc:1428] Setting key b954be1d6bcd9b8a710924663f1706c8
I20260925 01:27:52.213557 25892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: Bootstrap starting.
I20260925 01:27:52.216516 25892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:52.217451 25892 log.cc:824] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:52.220496 25892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: No bootstrap required, opened a new log
I20260925 01:27:52.222002 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:52.210898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.6.236.190:38395)
0925 01:27:52.211887 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:52.211899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:52.211940 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:27:52.213596 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:52.213621 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:52.214072 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:52.214455 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:52.214998 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.216228 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.217556 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.218565 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.218755 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:52.219854 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:52.219861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:52.219874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:52.220364 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:52.220438 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:52.220458 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:52.220542 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:52.220789 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 01:27:52.221102 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":344,"threads_started":1}
I20260925 01:27:52.228109 25892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } }
I20260925 01:27:52.228435 25892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:52.228533 25892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1222bcf9756f49e19260b684d449f81f, State: Initialized, Role: FOLLOWER
I20260925 01:27:52.229046 25892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } }
I20260925 01:27:52.229254 25892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:52.229373 25892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:52.229517 25892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:52.231598 25892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } }
I20260925 01:27:52.232062 25892 leader_election.cc:302] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1222bcf9756f49e19260b684d449f81f; no voters:
I20260925 01:27:52.232592 25892 leader_election.cc:288] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:52.232748 25897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:52.233937 25897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [term 1 LEADER]: Becoming Leader. State: Replica: 1222bcf9756f49e19260b684d449f81f, State: Running, Role: LEADER
I20260925 01:27:52.234426 25897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } }
I20260925 01:27:52.234661 25892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:52.236960 25898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1222bcf9756f49e19260b684d449f81f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } } }
I20260925 01:27:52.236922 25899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1222bcf9756f49e19260b684d449f81f. Latest consensus state: current_term: 1 leader_uuid: "1222bcf9756f49e19260b684d449f81f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1222bcf9756f49e19260b684d449f81f" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 38395 } } }
I20260925 01:27:52.237365 25899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:52.237370 25898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:52.239768 25906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:52.244721 25906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:52.252658 25906 catalog_manager.cc:1347] Generated new cluster ID: 3410f4530fd84dffb9fdfec793120ce2
I20260925 01:27:52.252770 25906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:52.263800 25906 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:52.265353 25906 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:52.271447 25906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: Generated new TSK 0
I20260925 01:27:52.272200 25906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:27:52.288798 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--builtin_ntp_servers=127.6.236.148:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:52.635494 25916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:52.635824 25916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:52.635947 25916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:52.643283 25916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:52.643383 25916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:52.643524 25916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:52.652550 25916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:52.653874 25916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:52.655670 25916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:52.668524 25921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:52.668594 25922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:52.668536 25924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:52.669384 25916 server_base.cc:1034] running on GCE node
I20260925 01:27:52.669970 25916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:52.670910 25916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:52.672117 25916 hybrid_clock.cc:648] HybridClock initialized: now 1790299672672052 us; error 48 us; skew 500 ppm
I20260925 01:27:52.672412 25916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:52.679911 25916 webserver.cc:460] Webserver started at http://127.6.236.129:38305/ using document root <none> and password file <none>
I20260925 01:27:52.680619 25916 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:52.680719 25916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:52.681028 25916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:52.683948 25916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "9aba678eac4441e8b84bd2b86f6e2950"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "a69e041922715605bb3c1850190688d2"
server_key_iv: "1df4e871f6347e9a36b5e83e96516ebe"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.684710 25916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "9aba678eac4441e8b84bd2b86f6e2950"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "a69e041922715605bb3c1850190688d2"
server_key_iv: "1df4e871f6347e9a36b5e83e96516ebe"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.689711 25916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:27:52.693281 25930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:52.694656 25916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:52.694815 25916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "9aba678eac4441e8b84bd2b86f6e2950"
format_stamp: "Formatted at 2026-09-25 01:27:52 on dist-test-slave-s0b2"
server_key: "a69e041922715605bb3c1850190688d2"
server_key_iv: "1df4e871f6347e9a36b5e83e96516ebe"
server_key_version: "encryptionkey@0"
I20260925 01:27:52.695025 25916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:52.723639 25916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:52.724478 25916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:52.724766 25916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:52.725716 25916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:52.727240 25916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:52.727340 25916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:52.727445 25916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:52.727512 25916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:52.756440 25916 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:38325
I20260925 01:27:52.756508 26042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:38325 every 8 connection(s)
I20260925 01:27:52.757680 25916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:52.765897 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 25916
I20260925 01:27:52.766055 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:52.766526 7090 external_mini_cluster.cc:1428] Setting key 8cb42e33085b7c2f9116327a332ca2f8
I20260925 01:27:52.769187 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--builtin_ntp_servers=127.6.236.148:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:27:52.769989 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:52.760162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:40267 (local address 127.6.236.190:38395)
0925 01:27:52.760599 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:52.760604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:52.761783 (+ 1179us) server_negotiation.cc:407] Connection header received
0925 01:27:52.763566 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:52.763571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:52.763658 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:52.763762 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:52.764841 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.765572 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.767299 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.767874 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.767998 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:52.769262 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:52.769267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:52.769270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:52.769286 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:52.769333 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:52.769347 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:52.769429 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:52.769651 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:27:52.769743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260925 01:27:52.771365 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:52.760771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38395 (local address 127.6.236.129:40267)
0925 01:27:52.761629 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:52.761664 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:52.763306 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:52.764004 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:52.764014 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:52.764123 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:52.764676 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:52.764693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.765720 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.765724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:52.767151 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:52.767165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:52.768065 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:52.768070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:52.768343 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:52.768968 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:52.768970 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:52.769010 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:52.769084 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:52.769439 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:52.769450 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:52.769458 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:27:52.769648 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 01:27:52.770026 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":248,"threads_started":1}
I20260925 01:27:52.773233 26043 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38395
I20260925 01:27:52.773545 26043 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:52.774405 26043 heartbeater.cc:511] Master 127.6.236.190:38395 requested a full tablet report, sending...
I20260925 01:27:52.776722 25857 ts_manager.cc:194] Registered new tserver with Master: 9aba678eac4441e8b84bd2b86f6e2950 (127.6.236.129:38325)
I20260925 01:27:52.778575 25857 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.129:40267
W20260925 01:27:53.083969 26047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:53.084230 26047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:53.084312 26047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:53.091302 26047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:53.091403 26047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:53.091504 26047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:53.100250 26047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:53.101414 26047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:53.103153 26047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:53.118613 26055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:53.118664 26052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:53.118736 26053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:53.119323 26047 server_base.cc:1034] running on GCE node
I20260925 01:27:53.119912 26047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:53.120795 26047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:53.121979 26047 hybrid_clock.cc:648] HybridClock initialized: now 1790299673121924 us; error 49 us; skew 500 ppm
I20260925 01:27:53.122303 26047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:53.124401 26047 webserver.cc:460] Webserver started at http://127.6.236.130:43517/ using document root <none> and password file <none>
I20260925 01:27:53.125056 26047 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:53.125149 26047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:53.125448 26047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:53.128222 26047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "4da636a2eed7404ba1f7e02ac4c59556"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "455318a44af9f85751e810a03f3d9c67"
server_key_iv: "04f2142d4284b99068ac21fded31ae64"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.128969 26047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "4da636a2eed7404ba1f7e02ac4c59556"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "455318a44af9f85751e810a03f3d9c67"
server_key_iv: "04f2142d4284b99068ac21fded31ae64"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.133682 26047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 01:27:53.137046 26061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.138443 26047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:53.138576 26047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "4da636a2eed7404ba1f7e02ac4c59556"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "455318a44af9f85751e810a03f3d9c67"
server_key_iv: "04f2142d4284b99068ac21fded31ae64"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.138772 26047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:53.156603 26047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:53.157390 26047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:53.157661 26047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:53.158605 26047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:53.160007 26047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:53.160099 26047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.160202 26047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:53.160275 26047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.189551 26047 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:34611
I20260925 01:27:53.189621 26173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:34611 every 8 connection(s)
I20260925 01:27:53.191109 26047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:53.193746 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26047
I20260925 01:27:53.193917 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:27:53.194527 7090 external_mini_cluster.cc:1428] Setting key 6f79328e60d3d27d7bc23a8a1517b64d
I20260925 01:27:53.197512 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--builtin_ntp_servers=127.6.236.148:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:27:53.206084 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.193834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:54407 (local address 127.6.236.190:38395)
0925 01:27:53.194434 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.194439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.195707 (+ 1268us) server_negotiation.cc:407] Connection header received
0925 01:27:53.198175 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.198182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.198296 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.198394 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:53.199753 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.200706 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.203000 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.203526 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.203641 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.205213 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:53.205219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:53.205223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:53.205244 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:53.205283 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:53.205298 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:53.205382 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.205691 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.205796 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472}
I20260925 01:27:53.207332 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.194553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38395 (local address 127.6.236.130:54407)
0925 01:27:53.195556 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.195601 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.197887 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.198638 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.198649 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.198763 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:53.199566 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.199590 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.200863 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.200869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.202855 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.202869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.203897 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.203903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.204157 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.204881 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:53.204884 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:53.204932 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:53.205026 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:53.205436 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:53.205447 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:53.205455 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.205650 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.206022 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":306,"threads_started":1}
I20260925 01:27:53.208622 26174 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38395
I20260925 01:27:53.208936 26174 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:53.209761 26174 heartbeater.cc:511] Master 127.6.236.190:38395 requested a full tablet report, sending...
I20260925 01:27:53.211370 25857 ts_manager.cc:194] Registered new tserver with Master: 4da636a2eed7404ba1f7e02ac4c59556 (127.6.236.130:34611)
I20260925 01:27:53.212481 25857 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.130:54407
W20260925 01:27:53.515506 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:53.515780 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:53.515913 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:53.522929 26177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:53.523027 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:53.523164 26177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:53.531924 26177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:38395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:53.533110 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:53.534909 26177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:53.547055 26182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:53.547125 26183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:53.547092 26185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:53.547823 26177 server_base.cc:1034] running on GCE node
I20260925 01:27:53.548404 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:53.549292 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:53.550479 26177 hybrid_clock.cc:648] HybridClock initialized: now 1790299673550430 us; error 37 us; skew 500 ppm
I20260925 01:27:53.550776 26177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:53.552840 26177 webserver.cc:460] Webserver started at http://127.6.236.131:38809/ using document root <none> and password file <none>
I20260925 01:27:53.553488 26177 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:53.553586 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:53.553885 26177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:53.556643 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "3a9955ea74c74e5b88be94a20e044832"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "65d54c9ab9a36081f7251dad4540f305"
server_key_iv: "dc223aa7652758ed580ab6cd24205946"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.557395 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "3a9955ea74c74e5b88be94a20e044832"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "65d54c9ab9a36081f7251dad4540f305"
server_key_iv: "dc223aa7652758ed580ab6cd24205946"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.562127 26177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 01:27:53.565501 26191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.566804 26177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:53.566946 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "3a9955ea74c74e5b88be94a20e044832"
format_stamp: "Formatted at 2026-09-25 01:27:53 on dist-test-slave-s0b2"
server_key: "65d54c9ab9a36081f7251dad4540f305"
server_key_iv: "dc223aa7652758ed580ab6cd24205946"
server_key_version: "encryptionkey@0"
I20260925 01:27:53.567138 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:53.585939 26177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:53.586742 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:53.587019 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:53.587931 26177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:53.589334 26177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:53.589427 26177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.589529 26177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:53.589597 26177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:53.618544 26177 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:44941
I20260925 01:27:53.618597 26303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:44941 every 8 connection(s)
I20260925 01:27:53.619791 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:53.625303 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26177
I20260925 01:27:53.625468 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:27:53.625990 7090 external_mini_cluster.cc:1428] Setting key 4fff66b093894aabdd0f37876f6ad92f
I20260925 01:27:53.631968 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.622375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:55887 (local address 127.6.236.190:38395)
0925 01:27:53.622598 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.622605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.623937 (+ 1332us) server_negotiation.cc:407] Connection header received
0925 01:27:53.625693 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.625699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.625784 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.625896 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:53.627074 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.628068 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.629745 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.630135 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.630225 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.631391 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:53.631396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:53.631399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:53.631415 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:53.631445 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:53.631455 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:53.631521 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.631673 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.631745 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:27:53.632863 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.622943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:38395 (local address 127.6.236.131:55887)
0925 01:27:53.623799 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.623833 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.625387 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.626054 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.626063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.626162 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:53.626892 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.626918 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.628210 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.628216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.629613 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.629625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.630324 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.630331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.630513 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.631121 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:53.631124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:53.631159 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:53.631236 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:53.631524 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:53.631530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:53.631535 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.631648 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.631922 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":266,"threads_started":1}
I20260925 01:27:53.634052 26304 heartbeater.cc:347] Connected to a master server at 127.6.236.190:38395
I20260925 01:27:53.634344 26304 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:53.635167 26304 heartbeater.cc:511] Master 127.6.236.190:38395 requested a full tablet report, sending...
I20260925 01:27:53.636535 25857 ts_manager.cc:194] Registered new tserver with Master: 3a9955ea74c74e5b88be94a20e044832 (127.6.236.131:44941)
I20260925 01:27:53.637287 25857 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.131:55887
I20260925 01:27:53.640605 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:53.649447 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.644383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.6.236.190:38395)
0925 01:27:53.644545 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.644549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.644959 (+ 410us) server_negotiation.cc:407] Connection header received
0925 01:27:53.645094 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.645097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.645161 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.645242 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:53.645644 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.646338 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.647593 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.647977 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.648058 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.648920 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:53.648925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:53.648928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:53.648942 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:53.648971 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:53.648982 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:53.649080 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.649203 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.649244 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:27:53.652665 25857 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:53.659113 25857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:53.680689 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.670337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:34611 (local address 127.0.0.1:48170)
0925 01:27:53.671062 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.671109 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.671333 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.674180 (+ 2847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.674185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.674209 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:53.674555 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.674567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.676350 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.676355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.677956 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.677968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.679801 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.679808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.680008 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.680064 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.680206 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.680376 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
I20260925 01:27:53.680778 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.670610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.6.236.130:34611)
0925 01:27:53.673654 (+ 3044us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.673662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.673688 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:27:53.673778 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.673785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.674002 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.674185 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:53.674776 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.676214 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.678145 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.679523 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.679663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.679940 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.680257 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.680414 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":2766,"threads_started":1}
I20260925 01:27:53.681252 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.671375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.6.236.129:38325)
0925 01:27:53.671917 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.671925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.672459 (+ 534us) server_negotiation.cc:407] Connection header received
0925 01:27:53.672748 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.672756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.673004 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.673159 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:53.674050 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.675587 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.678402 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.679774 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.679893 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.680098 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.680436 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.680595 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":285,"threads_started":1}
I20260925 01:27:53.681290 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.671428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44941 (local address 127.0.0.1:58158)
0925 01:27:53.673351 (+ 1923us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.673367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.673473 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.674508 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.674516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.674543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:53.674891 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.674900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.676758 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.676764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.678798 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.678812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.680657 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.680664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.680842 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.680872 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.680930 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.681040 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"mutex_wait_us":3,"thread_start_us":630,"threads_started":1}
I20260925 01:27:53.681978 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.670887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:44424)
0925 01:27:53.672351 (+ 1464us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.672368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.672562 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.673470 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.673477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.673505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:53.673904 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.673917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.675747 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.675752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.678134 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.678148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.680078 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.680084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.680246 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.680278 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.681501 (+ 1223us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.681645 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":66,"thread_start_us":2234,"threads_started":1}
I20260925 01:27:53.682251 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.672004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.6.236.131:44941)
0925 01:27:53.673991 (+ 1987us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.673999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.674028 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:27:53.674114 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.674120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.674345 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.674512 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:53.675203 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.676610 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.679005 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.680461 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.681607 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.681811 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.681861 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.681963 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":1698,"threads_started":1}
I20260925 01:27:53.684763 26109 tablet_service.cc:1467] Processing CreateTablet for tablet 26d00d2848f24b85b4ed07473bf3169a (DEFAULT_TABLE table=test-table [id=de1c8ff62d024343914e3354286809e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:53.685114 26239 tablet_service.cc:1467] Processing CreateTablet for tablet 26d00d2848f24b85b4ed07473bf3169a (DEFAULT_TABLE table=test-table [id=de1c8ff62d024343914e3354286809e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:53.685494 25978 tablet_service.cc:1467] Processing CreateTablet for tablet 26d00d2848f24b85b4ed07473bf3169a (DEFAULT_TABLE table=test-table [id=de1c8ff62d024343914e3354286809e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:53.686228 26239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d00d2848f24b85b4ed07473bf3169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:53.686219 26109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d00d2848f24b85b4ed07473bf3169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:53.686755 25978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d00d2848f24b85b4ed07473bf3169a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:53.697139 26322 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Bootstrap starting.
I20260925 01:27:53.700313 26324 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Bootstrap starting.
I20260925 01:27:53.700536 26322 tablet_bootstrap.cc:654] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:53.701942 26323 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Bootstrap starting.
I20260925 01:27:53.702334 26322 log.cc:824] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:53.704960 26324 tablet_bootstrap.cc:654] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:53.705155 26323 tablet_bootstrap.cc:654] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:53.705475 26322 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: No bootstrap required, opened a new log
I20260925 01:27:53.705744 26322 ts_tablet_manager.cc:1397] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:27:53.705991 26324 log.cc:824] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:53.706290 26323 log.cc:824] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:53.709108 26323 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: No bootstrap required, opened a new log
I20260925 01:27:53.709357 26323 ts_tablet_manager.cc:1397] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260925 01:27:53.715950 26324 tablet_bootstrap.cc:492] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: No bootstrap required, opened a new log
I20260925 01:27:53.715714 26322 raft_consensus.cc:348] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.716154 26322 raft_consensus.cc:374] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:53.716215 26324 ts_tablet_manager.cc:1397] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260925 01:27:53.716262 26322 raft_consensus.cc:729] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9955ea74c74e5b88be94a20e044832, State: Initialized, Role: FOLLOWER
I20260925 01:27:53.716877 26322 consensus_queue.cc:260] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.718005 26304 heartbeater.cc:503] Master 127.6.236.190:38395 was elected leader, sending a full tablet report...
I20260925 01:27:53.718633 26322 ts_tablet_manager.cc:1428] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260925 01:27:53.719126 26323 raft_consensus.cc:348] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.719511 26323 raft_consensus.cc:374] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:53.719604 26323 raft_consensus.cc:729] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da636a2eed7404ba1f7e02ac4c59556, State: Initialized, Role: FOLLOWER
I20260925 01:27:53.720165 26323 consensus_queue.cc:260] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.721608 26174 heartbeater.cc:503] Master 127.6.236.190:38395 was elected leader, sending a full tablet report...
I20260925 01:27:53.722220 26323 ts_tablet_manager.cc:1428] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260925 01:27:53.725425 26324 raft_consensus.cc:348] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.725801 26324 raft_consensus.cc:374] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:53.725893 26324 raft_consensus.cc:729] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aba678eac4441e8b84bd2b86f6e2950, State: Initialized, Role: FOLLOWER
I20260925 01:27:53.726447 26324 consensus_queue.cc:260] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.727362 26043 heartbeater.cc:503] Master 127.6.236.190:38395 was elected leader, sending a full tablet report...
I20260925 01:27:53.727754 26324 ts_tablet_manager.cc:1428] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
W20260925 01:27:53.759747 26044 tablet.cc:2369] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:53.788645 26329 raft_consensus.cc:482] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:53.788923 26329 raft_consensus.cc:504] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.790584 26329 leader_election.cc:288] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a9955ea74c74e5b88be94a20e044832 (127.6.236.131:44941), 9aba678eac4441e8b84bd2b86f6e2950 (127.6.236.129:38325)
I20260925 01:27:53.798594 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.790988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:58235 (local address 127.6.236.131:44941)
0925 01:27:53.791192 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.791197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.791934 (+ 737us) server_negotiation.cc:407] Connection header received
0925 01:27:53.792096 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.792099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.792159 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.792250 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:53.792845 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.793622 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.796441 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.797638 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.797797 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.797863 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.798183 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.798335 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:27:53.798596 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.791143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:58023 (local address 127.6.236.129:38325)
0925 01:27:53.791351 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.791358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.791861 (+ 503us) server_negotiation.cc:407] Connection header received
0925 01:27:53.792033 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.792039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.792122 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.792190 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:53.792783 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.793631 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.796441 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.797623 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.797791 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.797863 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.798192 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.798349 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:27:53.798758 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.790886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:44941 (local address 127.6.236.130:58235)
0925 01:27:53.791811 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.791829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.791964 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.792276 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.792281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.792304 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:53.792692 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.792704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.793761 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.793766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.796211 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.796226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.797805 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.797811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.797970 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.798072 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.798152 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.798390 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":253,"thread_start_us":525,"threads_started":1}
I20260925 01:27:53.799116 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.790882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.6.236.130:58023)
0925 01:27:53.791726 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:53.791746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:53.791880 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:53.792232 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:53.792236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:53.792261 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:53.792626 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.792639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.793769 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.793773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.796262 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:53.796273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.797779 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.797785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:53.797971 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.798063 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 01:27:53.798152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:27:53.798390 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":266,"thread_start_us":446,"threads_started":1}
I20260925 01:27:53.799711 26259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d00d2848f24b85b4ed07473bf3169a" candidate_uuid: "4da636a2eed7404ba1f7e02ac4c59556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9955ea74c74e5b88be94a20e044832" is_pre_election: true
I20260925 01:27:53.800184 26259 raft_consensus.cc:2454] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da636a2eed7404ba1f7e02ac4c59556 in term 0.
I20260925 01:27:53.800132 25998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d00d2848f24b85b4ed07473bf3169a" candidate_uuid: "4da636a2eed7404ba1f7e02ac4c59556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aba678eac4441e8b84bd2b86f6e2950" is_pre_election: true
I20260925 01:27:53.800583 25998 raft_consensus.cc:2454] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da636a2eed7404ba1f7e02ac4c59556 in term 0.
I20260925 01:27:53.800810 26062 leader_election.cc:302] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9955ea74c74e5b88be94a20e044832, 4da636a2eed7404ba1f7e02ac4c59556; no voters:
I20260925 01:27:53.801203 26329 raft_consensus.cc:2781] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:53.801333 26329 raft_consensus.cc:482] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:53.801429 26329 raft_consensus.cc:3037] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:53.803584 26329 raft_consensus.cc:504] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.804186 26329 leader_election.cc:288] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [CANDIDATE]: Term 1 election: Requested vote from peers 3a9955ea74c74e5b88be94a20e044832 (127.6.236.131:44941), 9aba678eac4441e8b84bd2b86f6e2950 (127.6.236.129:38325)
I20260925 01:27:53.804546 26259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d00d2848f24b85b4ed07473bf3169a" candidate_uuid: "4da636a2eed7404ba1f7e02ac4c59556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9955ea74c74e5b88be94a20e044832"
I20260925 01:27:53.804721 26259 raft_consensus.cc:3037] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:53.804663 25998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d00d2848f24b85b4ed07473bf3169a" candidate_uuid: "4da636a2eed7404ba1f7e02ac4c59556" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aba678eac4441e8b84bd2b86f6e2950"
I20260925 01:27:53.804864 25998 raft_consensus.cc:3037] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:53.806924 26259 raft_consensus.cc:2454] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da636a2eed7404ba1f7e02ac4c59556 in term 1.
I20260925 01:27:53.807029 25998 raft_consensus.cc:2454] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da636a2eed7404ba1f7e02ac4c59556 in term 1.
I20260925 01:27:53.807302 26062 leader_election.cc:302] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9955ea74c74e5b88be94a20e044832, 4da636a2eed7404ba1f7e02ac4c59556; no voters:
I20260925 01:27:53.807567 26329 raft_consensus.cc:2781] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:53.807972 26329 raft_consensus.cc:686] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 1 LEADER]: Becoming Leader. State: Replica: 4da636a2eed7404ba1f7e02ac4c59556, State: Running, Role: LEADER
I20260925 01:27:53.808410 26329 consensus_queue.cc:237] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } }
I20260925 01:27:53.812747 25856 catalog_manager.cc:5495] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da636a2eed7404ba1f7e02ac4c59556 (127.6.236.130). New cstate: current_term: 1 leader_uuid: "4da636a2eed7404ba1f7e02ac4c59556" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da636a2eed7404ba1f7e02ac4c59556" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 34611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260925 01:27:53.835122 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.829996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.6.236.130:34611)
0925 01:27:53.830174 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.830180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.830197 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:27:53.830438 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.830442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.830524 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.830647 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:53.831058 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.831748 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.833105 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.833516 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.833609 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.834050 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:53.834570 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:53.834695 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.834822 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.834861 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:27:53.842726 26259 raft_consensus.cc:1261] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 1 FOLLOWER]: Refusing update from remote peer 4da636a2eed7404ba1f7e02ac4c59556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:53.842736 25998 raft_consensus.cc:1261] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 1 FOLLOWER]: Refusing update from remote peer 4da636a2eed7404ba1f7e02ac4c59556: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:53.843678 26333 consensus_queue.cc:1035] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9955ea74c74e5b88be94a20e044832" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:53.843959 26329 consensus_queue.cc:1035] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aba678eac4441e8b84bd2b86f6e2950" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:53.855545 26338 mvcc.cc:205] Tried to move back new op lower bound from 7333067464050749440 to 7333067463923204096. Current Snapshot: MvccSnapshot[applied={T|T < 7333067464050749440}]
I20260925 01:27:53.855823 26337 mvcc.cc:205] Tried to move back new op lower bound from 7333067464050749440 to 7333067463923204096. Current Snapshot: MvccSnapshot[applied={T|T < 7333067464050749440}]
I20260925 01:27:53.872315 25856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:53.872561 25856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:53.875911 25856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1222bcf9756f49e19260b684d449f81f: Sending DeleteTablet for 3 replicas of tablet 26d00d2848f24b85b4ed07473bf3169a
I20260925 01:27:53.876941 26109 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d00d2848f24b85b4ed07473bf3169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:53 UTC) from {username='slave'} at 127.0.0.1:48170
I20260925 01:27:53.877009 26239 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d00d2848f24b85b4ed07473bf3169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:53 UTC) from {username='slave'} at 127.0.0.1:58158
I20260925 01:27:53.877668 25978 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d00d2848f24b85b4ed07473bf3169a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:53 UTC) from {username='slave'} at 127.0.0.1:44424
I20260925 01:27:53.878655 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25916
I20260925 01:27:53.900226 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:53.880382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.6.236.129:38325)
0925 01:27:53.880627 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:53.880634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:53.891290 (+ 10656us) server_negotiation.cc:407] Connection header received
0925 01:27:53.891748 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:53.891755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:53.891873 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:53.891960 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:53.892613 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.893837 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.895845 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:53.896545 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:53.896638 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:53.898323 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:53.898338 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:53.898358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:53.899190 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:53.899254 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:53.899281 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:53.899376 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:53.899685 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:27:53.899829 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:27:53.901403 26038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:53.907081 26347 tablet_replica.cc:330] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: stopping tablet replica
I20260925 01:27:53.907771 26347 raft_consensus.cc:2229] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:53.908396 26347 raft_consensus.cc:2258] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:53.909260 26346 tablet_replica.cc:330] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: stopping tablet replica
I20260925 01:27:53.909588 26346 raft_consensus.cc:2229] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:53.910115 26346 raft_consensus.cc:2258] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:53.912021 26346 ts_tablet_manager.cc:1905] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:53.912989 26347 ts_tablet_manager.cc:1905] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:53.919482 26346 ts_tablet_manager.cc:1918] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:53.919651 26346 log.cc:1196] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/26d00d2848f24b85b4ed07473bf3169a
I20260925 01:27:53.919716 26347 ts_tablet_manager.cc:1918] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:53.919873 26347 log.cc:1196] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/26d00d2848f24b85b4ed07473bf3169a
I20260925 01:27:53.920182 26346 ts_tablet_manager.cc:1939] T 26d00d2848f24b85b4ed07473bf3169a P 4da636a2eed7404ba1f7e02ac4c59556: Deleting consensus metadata
I20260925 01:27:53.920392 26347 ts_tablet_manager.cc:1939] T 26d00d2848f24b85b4ed07473bf3169a P 3a9955ea74c74e5b88be94a20e044832: Deleting consensus metadata
I20260925 01:27:53.922032 25841 catalog_manager.cc:4841] TS 4da636a2eed7404ba1f7e02ac4c59556 (127.6.236.130:34611): tablet 26d00d2848f24b85b4ed07473bf3169a (table test-table [id=de1c8ff62d024343914e3354286809e0]) successfully deleted
I20260925 01:27:53.922880 25841 catalog_manager.cc:4841] TS 3a9955ea74c74e5b88be94a20e044832 (127.6.236.131:44941): tablet 26d00d2848f24b85b4ed07473bf3169a (table test-table [id=de1c8ff62d024343914e3354286809e0]) successfully deleted
I20260925 01:27:53.998646 26348 tablet_replica.cc:330] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950: stopping tablet replica
I20260925 01:27:53.999169 26348 raft_consensus.cc:2229] T 26d00d2848f24b85b4ed07473bf3169a P 9aba678eac4441e8b84bd2b86f6e2950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:53.999320 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25916
W20260925 01:27:54.015848 25842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.236.129:38325 (error 108)
I20260925 01:27:54.015942 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26047
I20260925 01:27:54.017868 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.017410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36504)
0925 01:27:54.017580 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.017693 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 01:27:54.019138 25842 catalog_manager.cc:4584] TS 9aba678eac4441e8b84bd2b86f6e2950 (127.6.236.129:38325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26d00d2848f24b85b4ed07473bf3169a: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
I20260925 01:27:54.023217 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.017002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.6.236.130:34611)
0925 01:27:54.017173 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:54.017179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:54.017224 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:27:54.017301 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:54.017307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:54.017406 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:54.017511 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:54.018088 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:54.018877 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:54.020295 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:54.020705 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:54.020823 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:54.021833 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:54.021841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:54.021856 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:54.022515 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:54.022565 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:54.022586 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:54.022673 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:54.022813 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 01:27:54.022910 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:27:54.023897 26169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:54.072992 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.072369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36506)
0925 01:27:54.072674 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.072791 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260925 01:27:54.073932 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.073586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36520)
0925 01:27:54.073736 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.073809 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 01:27:54.132846 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26047
I20260925 01:27:54.134411 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.133820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36536)
0925 01:27:54.134096 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.134188 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 01:27:54.135320 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.135007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36552)
0925 01:27:54.135145 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.135216 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 01:27:54.148511 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26177
I20260925 01:27:54.155108 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.149223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.6.236.131:44941)
0925 01:27:54.149401 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:54.149407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:54.149427 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:27:54.149649 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:54.149653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:54.149762 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:54.149846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:54.150286 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:54.150997 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:54.152288 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:54.152659 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:54.152773 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:54.153768 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:54.153777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:54.153792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:54.154442 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:54.154492 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:54.154518 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:54.154613 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:54.154743 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 01:27:54.154845 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:27:54.155690 26299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:54.215832 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.215400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36564)
0925 01:27:54.215588 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.215690 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:27:54.216876 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:54.216498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:38325 (local address 127.0.0.1:36574)
0925 01:27:54.216688 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:54.216767 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 01:27:54.241434 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26177
I20260925 01:27:54.255263 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 25826
I20260925 01:27:54.256367 25887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:54.363250 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 25826
2026-09-25T01:27:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2638 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:27:54 dist-test-slave-s0b2 krb5kdc[26357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:27:54 dist-test-slave-s0b2 krb5kdc[26357](info): set up 2 sockets
Sep 25 01:27:54 dist-test-slave-s0b2 krb5kdc[26357](info): commencing operation
krb5kdc: starting...
W20260925 01:27:55.383162 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:27:55 dist-test-slave-s0b2 krb5kdc[26357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:27:55.612932 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35163
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:55.926700 26370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:55.926980 26370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:55.927110 26370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:55.934098 26370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:55.934190 26370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:55.934283 26370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:55.934363 26370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:55.934422 26370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:55.943859 26370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:35163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:55.945106 26370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:55.946380 26370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:55.957784 26378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:55.957775 26376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:55.958184 26375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:55.958540 26370 server_base.cc:1034] running on GCE node
I20260925 01:27:55.959242 26370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:55.960547 26370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:55.961764 26370 hybrid_clock.cc:648] HybridClock initialized: now 1790299675961689 us; error 45 us; skew 500 ppm
I20260925 01:27:55.962071 26370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:55 dist-test-slave-s0b2 krb5kdc[26357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:55.964967 26370 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:27:55.966344 26370 webserver.cc:460] Webserver started at http://127.6.236.190:37867/ using document root <none> and password file <none>
I20260925 01:27:55.966990 26370 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:55.967105 26370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:55.967391 26370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:55.970213 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "a44595bbca8345679c73dcb02952265b"
format_stamp: "Formatted at 2026-09-25 01:27:55 on dist-test-slave-s0b2"
server_key: "275fdc33be4fc8c7b0734870b5276954"
server_key_iv: "bad83561d74faf9d3299e82f939c5561"
server_key_version: "encryptionkey@0"
I20260925 01:27:55.970958 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "a44595bbca8345679c73dcb02952265b"
format_stamp: "Formatted at 2026-09-25 01:27:55 on dist-test-slave-s0b2"
server_key: "275fdc33be4fc8c7b0734870b5276954"
server_key_iv: "bad83561d74faf9d3299e82f939c5561"
server_key_version: "encryptionkey@0"
I20260925 01:27:55.975863 26370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:27:55.979107 26385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:55.980324 26370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:55.980479 26370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "a44595bbca8345679c73dcb02952265b"
format_stamp: "Formatted at 2026-09-25 01:27:55 on dist-test-slave-s0b2"
server_key: "275fdc33be4fc8c7b0734870b5276954"
server_key_iv: "bad83561d74faf9d3299e82f939c5561"
server_key_version: "encryptionkey@0"
I20260925 01:27:55.980651 26370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:56.029531 26370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:56.033459 26370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:56.033717 26370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:56.050688 26370 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:35163
I20260925 01:27:56.050724 26436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:35163 every 8 connection(s)
I20260925 01:27:56.052002 26370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:56.054731 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26370
I20260925 01:27:56.054910 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:56.055428 7090 external_mini_cluster.cc:1428] Setting key 0d75f6199465e2ed9a59625a9f0d437e
I20260925 01:27:56.056053 26437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:56.067929 26437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: Bootstrap starting.
Sep 25 01:27:56 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:56.070969 26437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:56.072117 26437 log.cc:824] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:56.074849 26437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: No bootstrap required, opened a new log
I20260925 01:27:56.077008 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:56.057371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.6.236.190:35163)
0925 01:27:56.058107 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:56.058119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:56.058159 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:27:56.060610 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:56.060632 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:56.061025 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:56.061398 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:56.062899 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.064154 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.065404 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.066136 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.066334 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:56.070162 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:56.070170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:56.070184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:56.070223 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:56.073244 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:56.073892 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:56.073899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:56.073904 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:56.074067 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:56.074473 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:56.074478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:56.074481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:56.074994 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:56.075277 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:56.075669 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 01:27:56.075975 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":323,"threads_started":1}
I20260925 01:27:56.082382 26437 raft_consensus.cc:348] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } }
I20260925 01:27:56.082719 26437 raft_consensus.cc:374] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:56.082811 26437 raft_consensus.cc:729] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44595bbca8345679c73dcb02952265b, State: Initialized, Role: FOLLOWER
I20260925 01:27:56.083298 26437 consensus_queue.cc:260] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } }
I20260925 01:27:56.083524 26437 raft_consensus.cc:388] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:56.083639 26437 raft_consensus.cc:482] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:56.083811 26437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:56.085803 26437 raft_consensus.cc:504] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } }
I20260925 01:27:56.086269 26437 leader_election.cc:302] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a44595bbca8345679c73dcb02952265b; no voters:
I20260925 01:27:56.086778 26437 leader_election.cc:288] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:56.086889 26442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:56.087466 26442 raft_consensus.cc:686] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [term 1 LEADER]: Becoming Leader. State: Replica: a44595bbca8345679c73dcb02952265b, State: Running, Role: LEADER
I20260925 01:27:56.087984 26442 consensus_queue.cc:237] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } }
I20260925 01:27:56.088577 26437 sys_catalog.cc:564] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:56.090888 26443 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a44595bbca8345679c73dcb02952265b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } } }
I20260925 01:27:56.091264 26443 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:56.091740 26444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a44595bbca8345679c73dcb02952265b. Latest consensus state: current_term: 1 leader_uuid: "a44595bbca8345679c73dcb02952265b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a44595bbca8345679c73dcb02952265b" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 35163 } } }
I20260925 01:27:56.092051 26444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:56.094296 26450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:56.098778 26450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:56.106184 26450 catalog_manager.cc:1347] Generated new cluster ID: 1fd9647f3e9a401489a14802926b961f
I20260925 01:27:56.106321 26450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:56.135362 26450 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:56.136622 26450 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:56.148473 26450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: Generated new TSK 0
I20260925 01:27:56.149287 26450 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.236.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
Entry for principal kudu/127.6.236.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab.
I20260925 01:27:56.209748 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:56.525172 26463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:56.525447 26463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:56.525586 26463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:56.532627 26463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:56.532747 26463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:56.532892 26463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:56.542513 26463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.129.keytab
--principal=kudu/127.6.236.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:56.543808 26463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:56.545076 26463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:56.557217 26471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:56.557343 26469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:56.557314 26468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:56.558082 26463 server_base.cc:1034] running on GCE node
I20260925 01:27:56.558667 26463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:56.559508 26463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:56.560734 26463 hybrid_clock.cc:648] HybridClock initialized: now 1790299676560681 us; error 41 us; skew 500 ppm
I20260925 01:27:56.561049 26463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:56 dist-test-slave-s0b2 krb5kdc[26357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:56.564069 26463 init.cc:377] Logged in from keytab as kudu/127.6.236.129@KRBTEST.COM (short username kudu)
I20260925 01:27:56.565378 26463 webserver.cc:460] Webserver started at http://127.6.236.129:40049/ using document root <none> and password file <none>
I20260925 01:27:56.566030 26463 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:56.566146 26463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:56.566469 26463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:56.569322 26463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "0fa7249a0c774c5b9b34c4471a958296"
format_stamp: "Formatted at 2026-09-25 01:27:56 on dist-test-slave-s0b2"
server_key: "1c87380bbbc238847f482606045591f6"
server_key_iv: "e90f4df6684545dc800a9898a04643ca"
server_key_version: "encryptionkey@0"
I20260925 01:27:56.570073 26463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "0fa7249a0c774c5b9b34c4471a958296"
format_stamp: "Formatted at 2026-09-25 01:27:56 on dist-test-slave-s0b2"
server_key: "1c87380bbbc238847f482606045591f6"
server_key_iv: "e90f4df6684545dc800a9898a04643ca"
server_key_version: "encryptionkey@0"
I20260925 01:27:56.575142 26463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:27:56.578984 26478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:56.580246 26463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:27:56.580385 26463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "0fa7249a0c774c5b9b34c4471a958296"
format_stamp: "Formatted at 2026-09-25 01:27:56 on dist-test-slave-s0b2"
server_key: "1c87380bbbc238847f482606045591f6"
server_key_iv: "e90f4df6684545dc800a9898a04643ca"
server_key_version: "encryptionkey@0"
I20260925 01:27:56.580564 26463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:56.593255 26463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:56.597084 26463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:56.597342 26463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:56.598196 26463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:56.599462 26463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:56.599535 26463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:56.599692 26463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:56.599756 26463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:56.628867 26463 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:39395
I20260925 01:27:56.628904 26590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:39395 every 8 connection(s)
I20260925 01:27:56.630085 26463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:56.635105 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26463
I20260925 01:27:56.635279 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:56.635811 7090 external_mini_cluster.cc:1428] Setting key 36ad122191e812ae55620c2c2e7fbbdc
Sep 25 01:27:56 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.129@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:56.651638 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:56.632675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:48663 (local address 127.6.236.190:35163)
0925 01:27:56.633131 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:56.633136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:56.634105 (+ 969us) server_negotiation.cc:407] Connection header received
0925 01:27:56.636434 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:56.636440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:56.636530 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:56.636639 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:56.638437 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.639150 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.641310 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.641715 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.641810 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:56.645717 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:56.645722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:56.645725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:56.645760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:56.648510 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:56.649231 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:56.649235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:56.649237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:56.649365 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:56.649791 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:56.649795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:56.649797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:56.650076 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:56.650209 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:56.651046 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 01:27:56.651277 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260925 01:27:56.652626 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:56.633225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35163 (local address 127.6.236.129:48663)
0925 01:27:56.633919 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:56.633957 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:56.636112 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:56.636896 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:56.636904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:56.637443 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:56.638233 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:56.638292 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.639311 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.639316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:56.641166 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:56.641182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:56.641918 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:56.641924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:56.642209 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:56.642896 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:56.642926 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:56.645481 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:56.648649 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:56.648656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:56.648670 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:56.649094 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:56.649476 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:56.649479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:56.649482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:56.649679 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:56.650200 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:56.650207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:56.650675 (+ 468us) client_negotiation.cc:769] Sending connection context
0925 01:27:56.650932 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 01:27:56.651474 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":277,"threads_started":1}
I20260925 01:27:56.654142 26591 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35163
I20260925 01:27:56.654446 26591 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:56.655140 26591 heartbeater.cc:511] Master 127.6.236.190:35163 requested a full tablet report, sending...
I20260925 01:27:56.657346 26402 ts_manager.cc:194] Registered new tserver with Master: 0fa7249a0c774c5b9b34c4471a958296 (127.6.236.129:39395)
I20260925 01:27:56.659377 26402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.129@KRBTEST.COM'} at 127.6.236.129:48663
WARNING: no policy specified for kudu/127.6.236.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
Entry for principal kudu/127.6.236.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab.
I20260925 01:27:56.689795 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:57.005390 26597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:57.005669 26597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:57.005790 26597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:57.013005 26597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:57.013111 26597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:57.013259 26597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:57.022847 26597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.130.keytab
--principal=kudu/127.6.236.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:57.024175 26597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:57.025483 26597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:57.038344 26602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:57.038357 26605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:57.038349 26603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:57.039404 26597 server_base.cc:1034] running on GCE node
I20260925 01:27:57.039983 26597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:57.040885 26597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:57.042104 26597 hybrid_clock.cc:648] HybridClock initialized: now 1790299677042045 us; error 48 us; skew 500 ppm
I20260925 01:27:57.042438 26597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:57 dist-test-slave-s0b2 krb5kdc[26357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:57.045349 26597 init.cc:377] Logged in from keytab as kudu/127.6.236.130@KRBTEST.COM (short username kudu)
I20260925 01:27:57.046761 26597 webserver.cc:460] Webserver started at http://127.6.236.130:46207/ using document root <none> and password file <none>
I20260925 01:27:57.047406 26597 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:57.047525 26597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:57.047813 26597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:57.050720 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "9938bcbdde6e40909e71025289bc0d7b"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "cda31fe8f6f9f430a46768ce7e2a4ed4"
server_key_iv: "a49d941201809bd5814f555c642b72da"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.051458 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "9938bcbdde6e40909e71025289bc0d7b"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "cda31fe8f6f9f430a46768ce7e2a4ed4"
server_key_iv: "a49d941201809bd5814f555c642b72da"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.056404 26597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:27:57.059811 26612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.061147 26597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:27:57.061316 26597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "9938bcbdde6e40909e71025289bc0d7b"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "cda31fe8f6f9f430a46768ce7e2a4ed4"
server_key_iv: "a49d941201809bd5814f555c642b72da"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.061489 26597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:57.074393 26597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:57.077953 26597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:57.078212 26597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:57.079180 26597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:57.080418 26597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:57.080495 26597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.080610 26597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:57.080667 26597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.109637 26597 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:37027
I20260925 01:27:57.109675 26724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:37027 every 8 connection(s)
I20260925 01:27:57.110862 26597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:27:57.111830 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26597
I20260925 01:27:57.111989 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:27:57.112478 7090 external_mini_cluster.cc:1428] Setting key e78935c2dcd3de1a8e4d42e4540064fe
Sep 25 01:27:57 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.130@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:57.132623 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.113390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:34869 (local address 127.6.236.190:35163)
0925 01:27:57.113600 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.113606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.114928 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 01:27:57.117363 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.117367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.117435 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.117518 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:57.119349 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.120067 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.122099 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.122536 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.122628 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.126392 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:57.126398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:57.126402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:57.126441 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:57.129416 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.130106 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.130110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.130113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.130294 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.130770 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.130774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.130776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.131172 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:57.131326 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.132101 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.132330 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:27:57.133592 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.113936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35163 (local address 127.6.236.130:34869)
0925 01:27:57.114761 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.114809 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.117086 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.117765 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.117776 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.118448 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:57.119180 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.119205 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.120202 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.120207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.121984 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.121998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.122745 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.122753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.123004 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.123645 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:57.123671 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:57.126162 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:57.129552 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:57.129558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:57.129572 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:57.129991 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:57.130416 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:57.130420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:57.130422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:57.130662 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:57.131310 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:57.131316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:57.131726 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.131966 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.132374 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":332,"threads_started":1}
I20260925 01:27:57.135005 26725 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35163
I20260925 01:27:57.135349 26725 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:57.136212 26725 heartbeater.cc:511] Master 127.6.236.190:35163 requested a full tablet report, sending...
I20260925 01:27:57.137868 26402 ts_manager.cc:194] Registered new tserver with Master: 9938bcbdde6e40909e71025289bc0d7b (127.6.236.130:37027)
I20260925 01:27:57.138710 26402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.130@KRBTEST.COM'} at 127.6.236.130:34869
WARNING: no policy specified for kudu/127.6.236.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
Entry for principal kudu/127.6.236.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab.
I20260925 01:27:57.169405 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:27:57.482592 26730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:57.482863 26730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:57.482956 26730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:57.490147 26730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:57.490270 26730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:57.490402 26730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:27:57.499982 26730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.131.keytab
--principal=kudu/127.6.236.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:35163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:57.501152 26730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:57.502476 26730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:57.514796 26735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:57.514822 26736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:57.515313 26738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:57.515997 26730 server_base.cc:1034] running on GCE node
I20260925 01:27:57.516530 26730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:57.517418 26730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:57.518618 26730 hybrid_clock.cc:648] HybridClock initialized: now 1790299677518563 us; error 47 us; skew 500 ppm
I20260925 01:27:57.518944 26730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:27:57 dist-test-slave-s0b2 krb5kdc[26357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:27:57.521982 26730 init.cc:377] Logged in from keytab as kudu/127.6.236.131@KRBTEST.COM (short username kudu)
I20260925 01:27:57.523416 26730 webserver.cc:460] Webserver started at http://127.6.236.131:40297/ using document root <none> and password file <none>
I20260925 01:27:57.524065 26730 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:57.524178 26730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:57.524466 26730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:57.527298 26730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "2bac1dbe8340442eb8098de44190f18e"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "75ce2bfd21190ac9b50a5a6366296c85"
server_key_iv: "2a8265d68db7a369447d75adaaaee00d"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.528014 26730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "2bac1dbe8340442eb8098de44190f18e"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "75ce2bfd21190ac9b50a5a6366296c85"
server_key_iv: "2a8265d68db7a369447d75adaaaee00d"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.532852 26730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:27:57.536099 26745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.537320 26730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 01:27:57.537456 26730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "2bac1dbe8340442eb8098de44190f18e"
format_stamp: "Formatted at 2026-09-25 01:27:57 on dist-test-slave-s0b2"
server_key: "75ce2bfd21190ac9b50a5a6366296c85"
server_key_iv: "2a8265d68db7a369447d75adaaaee00d"
server_key_version: "encryptionkey@0"
I20260925 01:27:57.537626 26730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:57.557492 26730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:57.561457 26730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:57.561722 26730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:57.562654 26730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:57.563938 26730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:57.564008 26730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.564124 26730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:57.564177 26730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:57.591409 26730 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:40435
I20260925 01:27:57.591447 26857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:40435 every 8 connection(s)
I20260925 01:27:57.592643 26730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:27:57.601358 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26730
I20260925 01:27:57.601521 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:27:57.601958 7090 external_mini_cluster.cc:1428] Setting key 5fe401d70b3320e39f2070494c0346af
Sep 25 01:27:57 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.131@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:27:57.611254 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.595165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:52025 (local address 127.6.236.190:35163)
0925 01:27:57.595341 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.595346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.596499 (+ 1153us) server_negotiation.cc:407] Connection header received
0925 01:27:57.598213 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.598217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.598319 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.598405 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:57.599802 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.600585 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.602142 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.602831 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.602957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.606604 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:57.606609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:57.606612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:57.606643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:57.608786 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.609388 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.609392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.609394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.609506 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.609835 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.609838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.609840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.610101 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:57.610229 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.610841 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.611010 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:27:57.612164 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.595651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:35163 (local address 127.6.236.131:52025)
0925 01:27:57.596342 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.596380 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.597957 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.598600 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.598608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.599100 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:57.599650 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.599666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.600708 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.600711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.602004 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.602017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.603055 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.603064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.603358 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.604022 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:27:57.604059 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:57.606412 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:57.608898 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:57.608904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:57.608916 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:57.609303 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:57.609580 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:27:57.609583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:27:57.609585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:27:57.609757 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:27:57.610187 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:57.610192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:57.610522 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.610770 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.611132 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260925 01:27:57.613371 26858 heartbeater.cc:347] Connected to a master server at 127.6.236.190:35163
I20260925 01:27:57.613626 26858 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:57.614316 26858 heartbeater.cc:511] Master 127.6.236.190:35163 requested a full tablet report, sending...
I20260925 01:27:57.615711 26402 ts_manager.cc:194] Registered new tserver with Master: 2bac1dbe8340442eb8098de44190f18e (127.6.236.131:40435)
I20260925 01:27:57.616706 26402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.236.131@KRBTEST.COM'} at 127.6.236.131:52025
I20260925 01:27:57.617218 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:27:57.633275 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.621173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.6.236.190:35163)
0925 01:27:57.621363 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.621369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.621708 (+ 339us) server_negotiation.cc:407] Connection header received
0925 01:27:57.621864 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.621868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.621927 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.622005 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:57.623377 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.624094 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.625367 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.625802 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.625912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.628252 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:57.628257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:57.628260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:57.628290 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:57.630763 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.631349 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.631354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.631356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.631473 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:57.631822 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:57.631826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:57.631829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:57.632203 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:57.632436 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.632773 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.632979 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:27:57.636811 26402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:27:57.643416 26402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:27:57.662436 26591 heartbeater.cc:503] Master 127.6.236.190:35163 was elected leader, sending a full tablet report...
I20260925 01:27:57.681186 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.655049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.6.236.130:37027)
0925 01:27:57.673267 (+ 18218us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.673275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.673304 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:27:57.673400 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.673408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.673697 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.673885 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:57.674822 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.676388 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.679066 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.680087 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.680212 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.680429 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.680721 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.680891 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18068,"thread_start_us":17977,"threads_started":1}
I20260925 01:27:57.681363 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.654899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37027 (local address 127.0.0.1:42334)
0925 01:27:57.669953 (+ 15054us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.670002 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.670202 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.674168 (+ 3966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.674176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.674207 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:57.674615 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.674628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.676537 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.676544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.678898 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.678914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.680290 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.680297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.680503 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.680540 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.680645 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.680915 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14763,"mutex_wait_us":54,"thread_start_us":15489,"threads_started":1}
I20260925 01:27:57.682121 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.655635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:40435 (local address 127.0.0.1:51184)
0925 01:27:57.671130 (+ 15495us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.671147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.671255 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.674883 (+ 3628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.674887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.674912 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:57.675315 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.675328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.677224 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.677228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.679242 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.679256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.681297 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.681304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.681475 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.681508 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.681603 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.681726 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15367,"mutex_wait_us":1,"thread_start_us":15219,"threads_started":1}
I20260925 01:27:57.682329 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.655956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.6.236.131:40435)
0925 01:27:57.670710 (+ 14754us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.670719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.673456 (+ 2737us) server_negotiation.cc:407] Connection header received
0925 01:27:57.673678 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.673684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.673913 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.674084 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:57.675591 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.677047 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.679399 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.680786 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.680909 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.681140 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.681824 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.681986 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14581,"thread_start_us":17289,"threads_started":1}
I20260925 01:27:57.685243 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.671403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:39395 (local address 127.0.0.1:51608)
0925 01:27:57.676997 (+ 5594us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.677015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.677128 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.677999 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.678003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.678031 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:57.678484 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.678497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.680276 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.680281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.682460 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.682475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.684532 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.684538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.684680 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.684703 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.684792 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.684906 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5434,"mutex_wait_us":42,"thread_start_us":5273,"threads_started":1}
I20260925 01:27:57.685513 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.671422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.6.236.129:39395)
0925 01:27:57.674813 (+ 3391us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.674821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.677258 (+ 2437us) server_negotiation.cc:407] Connection header received
0925 01:27:57.677360 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.677367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.677589 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.677765 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:57.678697 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.680104 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.682723 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.683994 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.684119 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.684338 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.684888 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.685050 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3206,"thread_start_us":3040,"threads_started":1}
I20260925 01:27:57.685546 26660 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffcecfba1c94f51af517df951a48e05 (DEFAULT_TABLE table=test-table [id=8479c48fe47143e681c427dc84df2745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:57.686165 26793 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffcecfba1c94f51af517df951a48e05 (DEFAULT_TABLE table=test-table [id=8479c48fe47143e681c427dc84df2745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:57.687124 26660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffcecfba1c94f51af517df951a48e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:57.689379 26793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffcecfba1c94f51af517df951a48e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:57.689388 26526 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffcecfba1c94f51af517df951a48e05 (DEFAULT_TABLE table=test-table [id=8479c48fe47143e681c427dc84df2745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:27:57.690678 26526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffcecfba1c94f51af517df951a48e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:57.701328 26877 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Bootstrap starting.
I20260925 01:27:57.703227 26876 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Bootstrap starting.
I20260925 01:27:57.704373 26878 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Bootstrap starting.
I20260925 01:27:57.706357 26876 tablet_bootstrap.cc:654] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:57.706358 26877 tablet_bootstrap.cc:654] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:57.707317 26876 log.cc:824] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:57.707547 26877 log.cc:824] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:57.707760 26878 tablet_bootstrap.cc:654] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:57.708915 26878 log.cc:824] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:57.710456 26877 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: No bootstrap required, opened a new log
I20260925 01:27:57.710706 26877 ts_tablet_manager.cc:1397] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:27:57.710876 26876 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: No bootstrap required, opened a new log
I20260925 01:27:57.711160 26876 ts_tablet_manager.cc:1397] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 01:27:57.716671 26878 tablet_bootstrap.cc:492] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: No bootstrap required, opened a new log
I20260925 01:27:57.716935 26878 ts_tablet_manager.cc:1397] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260925 01:27:57.717805 26877 raft_consensus.cc:348] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.718381 26877 raft_consensus.cc:374] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:57.718531 26877 raft_consensus.cc:729] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9938bcbdde6e40909e71025289bc0d7b, State: Initialized, Role: FOLLOWER
I20260925 01:27:57.719162 26877 consensus_queue.cc:260] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.720939 26876 raft_consensus.cc:348] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.721361 26876 raft_consensus.cc:374] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:57.721513 26876 raft_consensus.cc:729] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bac1dbe8340442eb8098de44190f18e, State: Initialized, Role: FOLLOWER
I20260925 01:27:57.722009 26725 heartbeater.cc:503] Master 127.6.236.190:35163 was elected leader, sending a full tablet report...
I20260925 01:27:57.722285 26876 consensus_queue.cc:260] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.722620 26877 ts_tablet_manager.cc:1428] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260925 01:27:57.726068 26858 heartbeater.cc:503] Master 127.6.236.190:35163 was elected leader, sending a full tablet report...
I20260925 01:27:57.726827 26876 ts_tablet_manager.cc:1428] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260925 01:27:57.729416 26878 raft_consensus.cc:348] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.729739 26878 raft_consensus.cc:374] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:57.729827 26878 raft_consensus.cc:729] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa7249a0c774c5b9b34c4471a958296, State: Initialized, Role: FOLLOWER
I20260925 01:27:57.730397 26878 consensus_queue.cc:260] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.731837 26878 ts_tablet_manager.cc:1428] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260925 01:27:57.844256 26859 tablet.cc:2369] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:57.862818 26726 tablet.cc:2369] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:27:57.882424 26592 tablet.cc:2369] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:27:57.963101 26884 raft_consensus.cc:482] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:27:57.963387 26884 raft_consensus.cc:504] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.965059 26884 leader_election.cc:288] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9938bcbdde6e40909e71025289bc0d7b (127.6.236.130:37027), 2bac1dbe8340442eb8098de44190f18e (127.6.236.131:40435)
I20260925 01:27:57.971930 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.965677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:55925 (local address 127.6.236.130:37027)
0925 01:27:57.965877 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.965881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.966365 (+ 484us) server_negotiation.cc:407] Connection header received
0925 01:27:57.966624 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.966630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.966710 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.966816 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:57.967394 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.968223 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.970155 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.970982 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.971092 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.971144 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.971552 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.971684 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:27:57.971959 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.965582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:59743 (local address 127.6.236.131:40435)
0925 01:27:57.965782 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:57.965788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:57.966943 (+ 1155us) server_negotiation.cc:407] Connection header received
0925 01:27:57.967121 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:57.967126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:57.967203 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:57.967298 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:27:57.967810 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.968629 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.970340 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.971085 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.971233 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.971300 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:57.971612 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:27:57.971742 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:27:57.972040 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.965505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37027 (local address 127.6.236.129:55925)
0925 01:27:57.966257 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.966287 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.966457 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.966846 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.966850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.966873 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:57.967249 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.967263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.968349 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.968355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.969968 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.969981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.971177 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.971184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.971369 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.971429 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.971499 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.971614 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":73,"thread_start_us":406,"threads_started":1}
I20260925 01:27:57.972090 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:57.965597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:40435 (local address 127.6.236.129:59743)
0925 01:27:57.966840 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:57.966855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:57.966993 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:57.967320 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:57.967324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:57.967349 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:27:57.967669 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.967680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.968756 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.968760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.970174 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:57.970187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:57.971287 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:57.971292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:57.971455 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:57.971479 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:27:57.971566 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 01:27:57.971683 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":34,"thread_start_us":892,"threads_started":1}
I20260925 01:27:57.973143 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffcecfba1c94f51af517df951a48e05" candidate_uuid: "0fa7249a0c774c5b9b34c4471a958296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9938bcbdde6e40909e71025289bc0d7b" is_pre_election: true
I20260925 01:27:57.973168 26813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffcecfba1c94f51af517df951a48e05" candidate_uuid: "0fa7249a0c774c5b9b34c4471a958296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bac1dbe8340442eb8098de44190f18e" is_pre_election: true
I20260925 01:27:57.973594 26680 raft_consensus.cc:2454] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa7249a0c774c5b9b34c4471a958296 in term 0.
I20260925 01:27:57.973594 26813 raft_consensus.cc:2454] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa7249a0c774c5b9b34c4471a958296 in term 0.
I20260925 01:27:57.974197 26482 leader_election.cc:302] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7249a0c774c5b9b34c4471a958296, 9938bcbdde6e40909e71025289bc0d7b; no voters:
I20260925 01:27:57.974611 26884 raft_consensus.cc:2781] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:27:57.974738 26884 raft_consensus.cc:482] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:27:57.974833 26884 raft_consensus.cc:3037] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:57.976874 26884 raft_consensus.cc:504] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.977407 26884 leader_election.cc:288] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [CANDIDATE]: Term 1 election: Requested vote from peers 9938bcbdde6e40909e71025289bc0d7b (127.6.236.130:37027), 2bac1dbe8340442eb8098de44190f18e (127.6.236.131:40435)
I20260925 01:27:57.977808 26813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffcecfba1c94f51af517df951a48e05" candidate_uuid: "0fa7249a0c774c5b9b34c4471a958296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bac1dbe8340442eb8098de44190f18e"
I20260925 01:27:57.977804 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffcecfba1c94f51af517df951a48e05" candidate_uuid: "0fa7249a0c774c5b9b34c4471a958296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9938bcbdde6e40909e71025289bc0d7b"
I20260925 01:27:57.977969 26813 raft_consensus.cc:3037] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:57.977993 26680 raft_consensus.cc:3037] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:57.980031 26680 raft_consensus.cc:2454] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa7249a0c774c5b9b34c4471a958296 in term 1.
I20260925 01:27:57.980033 26813 raft_consensus.cc:2454] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa7249a0c774c5b9b34c4471a958296 in term 1.
I20260925 01:27:57.980410 26480 leader_election.cc:302] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa7249a0c774c5b9b34c4471a958296, 2bac1dbe8340442eb8098de44190f18e; no voters:
I20260925 01:27:57.980662 26884 raft_consensus.cc:2781] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:57.981065 26884 raft_consensus.cc:686] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa7249a0c774c5b9b34c4471a958296, State: Running, Role: LEADER
I20260925 01:27:57.981570 26884 consensus_queue.cc:237] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } }
I20260925 01:27:57.985749 26401 catalog_manager.cc:5495] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa7249a0c774c5b9b34c4471a958296 (127.6.236.129). New cstate: current_term: 1 leader_uuid: "0fa7249a0c774c5b9b34c4471a958296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa7249a0c774c5b9b34c4471a958296" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 39395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 } health_report { overall_health: UNKNOWN } } }
I20260925 01:27:58.033008 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:58.026262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.6.236.129:39395)
0925 01:27:58.026876 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:58.026883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:58.026904 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:27:58.027021 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:58.027027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:58.027130 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:58.027300 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:27:58.027769 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.028804 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.030609 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.031068 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.031178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:58.031756 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:27:58.032486 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:27:58.032674 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:58.032742 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 01:27:58.032811 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457}
I20260925 01:27:58.043232 26680 raft_consensus.cc:1261] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 1 FOLLOWER]: Refusing update from remote peer 0fa7249a0c774c5b9b34c4471a958296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:58.043232 26813 raft_consensus.cc:1261] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 1 FOLLOWER]: Refusing update from remote peer 0fa7249a0c774c5b9b34c4471a958296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:27:58.044260 26887 consensus_queue.cc:1035] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9938bcbdde6e40909e71025289bc0d7b" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:58.044682 26884 consensus_queue.cc:1035] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bac1dbe8340442eb8098de44190f18e" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:27:58.058693 26890 mvcc.cc:205] Tried to move back new op lower bound from 7333067481254490112 to 7333067481015951360. Current Snapshot: MvccSnapshot[applied={T|T < 7333067481254490112}]
I20260925 01:27:58.072978 26891 mvcc.cc:205] Tried to move back new op lower bound from 7333067481254490112 to 7333067481015951360. Current Snapshot: MvccSnapshot[applied={T|T < 7333067481254490112}]
I20260925 01:27:58.088047 26401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:58.088308 26401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:27:58.091765 26401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a44595bbca8345679c73dcb02952265b: Sending DeleteTablet for 3 replicas of tablet 5ffcecfba1c94f51af517df951a48e05
I20260925 01:27:58.092821 26660 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffcecfba1c94f51af517df951a48e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:58 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:42334
I20260925 01:27:58.092835 26526 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffcecfba1c94f51af517df951a48e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:58 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:51608
I20260925 01:27:58.092940 26793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffcecfba1c94f51af517df951a48e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:27:58 UTC) from {username='kudu', principal='kudu/127.6.236.190@KRBTEST.COM'} at 127.0.0.1:51184
I20260925 01:27:58.093791 26902 tablet_replica.cc:330] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: stopping tablet replica
I20260925 01:27:58.094295 26902 raft_consensus.cc:2229] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:27:58.094633 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26463
I20260925 01:27:58.094897 26902 raft_consensus.cc:2258] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:58.096954 26902 ts_tablet_manager.cc:1905] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:58.103857 26902 ts_tablet_manager.cc:1918] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:58.104048 26902 log.cc:1196] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/5ffcecfba1c94f51af517df951a48e05
I20260925 01:27:58.104579 26902 ts_tablet_manager.cc:1939] T 5ffcecfba1c94f51af517df951a48e05 P 2bac1dbe8340442eb8098de44190f18e: Deleting consensus metadata
I20260925 01:27:58.105391 26901 tablet_replica.cc:330] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: stopping tablet replica
I20260925 01:27:58.106403 26387 catalog_manager.cc:4841] TS 2bac1dbe8340442eb8098de44190f18e (127.6.236.131:40435): tablet 5ffcecfba1c94f51af517df951a48e05 (table test-table [id=8479c48fe47143e681c427dc84df2745]) successfully deleted
I20260925 01:27:58.106832 26901 raft_consensus.cc:2229] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:27:58.107620 26901 raft_consensus.cc:2258] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:58.111619 26901 ts_tablet_manager.cc:1905] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:58.112603 26900 tablet_replica.cc:330] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: stopping tablet replica
I20260925 01:27:58.113116 26900 raft_consensus.cc:2229] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 25 01:27:58 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.129@KRBTEST.COM
I20260925 01:27:58.120371 26901 ts_tablet_manager.cc:1918] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:58.120921 26900 raft_consensus.cc:2258] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:27:58.122961 26900 ts_tablet_manager.cc:1905] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:27:58.128921 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:58.095587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.6.236.129:39395)
0925 01:27:58.095743 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:58.095748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:58.109556 (+ 13808us) server_negotiation.cc:407] Connection header received
0925 01:27:58.110010 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:58.110015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:58.110109 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:58.110187 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:58.112382 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.113538 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.115393 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.116054 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.116145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:58.121726 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:58.121740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:58.121757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:58.121814 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:58.125528 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.126374 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.126383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.126386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.126562 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.127051 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.127055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.127058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.127477 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:58.127749 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:58.128323 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 01:27:58.128568 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 01:27:58.129879 26900 ts_tablet_manager.cc:1918] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:27:58.121501 26901 log.cc:1196] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/5ffcecfba1c94f51af517df951a48e05
I20260925 01:27:58.130604 26901 ts_tablet_manager.cc:1939] T 5ffcecfba1c94f51af517df951a48e05 P 0fa7249a0c774c5b9b34c4471a958296: Deleting consensus metadata
I20260925 01:27:58.130980 26586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:58.131681 26900 log.cc:1196] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/5ffcecfba1c94f51af517df951a48e05
I20260925 01:27:58.132215 26900 ts_tablet_manager.cc:1939] T 5ffcecfba1c94f51af517df951a48e05 P 9938bcbdde6e40909e71025289bc0d7b: Deleting consensus metadata
I20260925 01:27:58.133955 26389 catalog_manager.cc:4841] TS 9938bcbdde6e40909e71025289bc0d7b (127.6.236.130:37027): tablet 5ffcecfba1c94f51af517df951a48e05 (table test-table [id=8479c48fe47143e681c427dc84df2745]) successfully deleted
I20260925 01:27:58.230605 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26463
I20260925 01:27:58.230851 26389 catalog_manager.cc:4841] TS 0fa7249a0c774c5b9b34c4471a958296 (127.6.236.129:39395): tablet 5ffcecfba1c94f51af517df951a48e05 (table test-table [id=8479c48fe47143e681c427dc84df2745]) successfully deleted
I20260925 01:27:58.246526 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26597
Sep 25 01:27:58 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.130@KRBTEST.COM
I20260925 01:27:58.260210 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:58.247255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.6.236.130:37027)
0925 01:27:58.247438 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:58.247443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:58.247463 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:27:58.247661 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:58.247665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:58.247739 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:58.247822 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:58.249051 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.249751 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.251043 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.251434 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.251542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:58.255473 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:58.255487 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:58.255502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:58.255540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:58.257988 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.258610 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.258618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.258620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.258762 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.259034 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.259037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.259039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.259375 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:58.259581 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:58.259783 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 01:27:58.259946 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:27:58.260830 26720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:58.348302 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26597
I20260925 01:27:58.362591 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26730
Sep 25 01:27:58 dist-test-slave-s0b2 krb5kdc[26357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.131@KRBTEST.COM
I20260925 01:27:58.376295 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:58.363334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.6.236.131:40435)
0925 01:27:58.363492 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:58.363496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:58.363515 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:27:58.363649 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:58.363652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:58.363738 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:58.363823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:58.364987 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.365676 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.366919 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:58.367293 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:58.367383 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:58.371535 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:58.371550 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:58.371567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:27:58.371598 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:58.374018 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.374661 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.374669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.374671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.374823 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:27:58.375097 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:27:58.375100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:27:58.375102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:27:58.375425 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:58.375635 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:58.375872 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 01:27:58.376035 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:27:58.376941 26853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:58.463513 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26730
I20260925 01:27:58.477317 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26370
I20260925 01:27:58.478364 26432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:27:58.586704 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26370
2026-09-25T01:27:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4220 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T01:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:27:58Z Disabled control of system clock
I20260925 01:27:58.658053 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37223
--webserver_interface=127.6.236.190
--webserver_port=0
--builtin_ntp_servers=127.6.236.148:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:58.974444 26911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:58.974737 26911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:58.974859 26911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:58.981998 26911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:27:58.982108 26911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:58.982177 26911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:58.982271 26911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:27:58.982331 26911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:27:58.991602 26911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:58.992852 26911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:58.994643 26911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:59.005394 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.005467 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.005425 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:59.006214 26911 server_base.cc:1034] running on GCE node
I20260925 01:27:59.006961 26911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:59.008246 26911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:59.009443 26911 hybrid_clock.cc:648] HybridClock initialized: now 1790299679009394 us; error 42 us; skew 500 ppm
I20260925 01:27:59.009786 26911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:59.011931 26911 webserver.cc:460] Webserver started at http://127.6.236.190:40817/ using document root <none> and password file <none>
I20260925 01:27:59.012564 26911 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:59.012672 26911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:59.012979 26911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:59.015748 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "1d61c817804149618142ec6c3f62aa50"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "18beb5b4d2da4d499d876d5fb4f8bd2c"
server_key_iv: "f124011fba06fbc058ced87b32383a81"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.016484 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "1d61c817804149618142ec6c3f62aa50"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "18beb5b4d2da4d499d876d5fb4f8bd2c"
server_key_iv: "f124011fba06fbc058ced87b32383a81"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.021353 26911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:27:59.024694 26925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:59.025960 26911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:27:59.026124 26911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "1d61c817804149618142ec6c3f62aa50"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "18beb5b4d2da4d499d876d5fb4f8bd2c"
server_key_iv: "f124011fba06fbc058ced87b32383a81"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.026351 26911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:59.044242 26911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:59.045130 26911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:59.045400 26911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:59.062175 26911 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:37223
I20260925 01:27:59.062198 26976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:37223 every 8 connection(s)
I20260925 01:27:59.063483 26911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:27:59.067116 26977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:27:59.067574 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 26911
I20260925 01:27:59.067739 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:27:59.068140 7090 external_mini_cluster.cc:1428] Setting key 32949f9ef8f06763b7ad47759ed29706
I20260925 01:27:59.080083 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: Bootstrap starting.
I20260925 01:27:59.081629 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:59.069888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.6.236.190:37223)
0925 01:27:59.070878 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:59.070892 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:59.070933 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:27:59.072887 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:59.072909 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:59.073400 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:59.073720 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:59.074302 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.075532 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.077427 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.078105 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.078259 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:59.079463 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:59.079471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:59.079484 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:59.079991 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:59.080066 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:59.080090 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:59.080177 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:59.080407 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:27:59.080720 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":387,"threads_started":1}
I20260925 01:27:59.083324 26977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: Neither blocks nor log segments found. Creating new log.
I20260925 01:27:59.084297 26977 log.cc:824] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: Log is configured to *not* fsync() on all Append() calls
I20260925 01:27:59.086752 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: No bootstrap required, opened a new log
I20260925 01:27:59.093637 26977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } }
I20260925 01:27:59.093992 26977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:27:59.094092 26977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d61c817804149618142ec6c3f62aa50, State: Initialized, Role: FOLLOWER
I20260925 01:27:59.094645 26977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } }
I20260925 01:27:59.094859 26977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:27:59.094982 26977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:27:59.095132 26977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:27:59.097213 26977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } }
I20260925 01:27:59.097687 26977 leader_election.cc:302] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d61c817804149618142ec6c3f62aa50; no voters:
I20260925 01:27:59.098232 26977 leader_election.cc:288] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:27:59.098412 26982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:27:59.099028 26982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [term 1 LEADER]: Becoming Leader. State: Replica: 1d61c817804149618142ec6c3f62aa50, State: Running, Role: LEADER
I20260925 01:27:59.099506 26982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } }
I20260925 01:27:59.100224 26977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:27:59.102080 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d61c817804149618142ec6c3f62aa50. Latest consensus state: current_term: 1 leader_uuid: "1d61c817804149618142ec6c3f62aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } } }
I20260925 01:27:59.102530 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:59.102108 26983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d61c817804149618142ec6c3f62aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d61c817804149618142ec6c3f62aa50" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37223 } } }
I20260925 01:27:59.103051 26983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50 [sys.catalog]: This master's current role is: LEADER
I20260925 01:27:59.104339 26988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:27:59.109669 26988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:27:59.118224 26988 catalog_manager.cc:1347] Generated new cluster ID: e5ac72b0e3e34e27af3132f06b285ecd
I20260925 01:27:59.118353 26988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:27:59.129803 26988 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:27:59.131225 26988 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:27:59.137883 26988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: Generated new TSK 0
I20260925 01:27:59.138736 26988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:27:59.154369 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.129:0
--local_ip_for_outbound_sockets=127.6.236.129
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--builtin_ntp_servers=127.6.236.148:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:27:59.478744 27001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:59.479014 27001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:59.479125 27001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:59.486164 27001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:59.486280 27001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:59.486430 27001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.129
I20260925 01:27:59.495344 27001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.236.129
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.129
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:59.496552 27001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:59.498359 27001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:59.510726 27006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.510766 27009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.510789 27007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:59.511582 27001 server_base.cc:1034] running on GCE node
I20260925 01:27:59.512149 27001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:59.513067 27001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:59.514317 27001 hybrid_clock.cc:648] HybridClock initialized: now 1790299679514263 us; error 42 us; skew 500 ppm
I20260925 01:27:59.514621 27001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:59.516722 27001 webserver.cc:460] Webserver started at http://127.6.236.129:34453/ using document root <none> and password file <none>
I20260925 01:27:59.517385 27001 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:59.517485 27001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:59.517786 27001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:59.520668 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data/instance:
uuid: "fe861122cc644c7ca4aaa4e4b8b032c3"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "4a6e339939741af373685a72048997b1"
server_key_iv: "d0e397672d1e2d0b7d9a5d564d7bfdab"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.521430 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance:
uuid: "fe861122cc644c7ca4aaa4e4b8b032c3"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "4a6e339939741af373685a72048997b1"
server_key_iv: "d0e397672d1e2d0b7d9a5d564d7bfdab"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.526371 27001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:27:59.529618 27015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:59.531018 27001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:27:59.531172 27001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal
uuid: "fe861122cc644c7ca4aaa4e4b8b032c3"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "4a6e339939741af373685a72048997b1"
server_key_iv: "d0e397672d1e2d0b7d9a5d564d7bfdab"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.531373 27001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:59.549612 27001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:59.550467 27001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:59.550737 27001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:59.551635 27001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:27:59.553006 27001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:27:59.553099 27001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:59.553221 27001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:27:59.553290 27001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:59.582177 27001 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.129:45913
I20260925 01:27:59.582216 27127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.129:45913 every 8 connection(s)
I20260925 01:27:59.583464 27001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/data/info.pb
I20260925 01:27:59.590895 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27001
I20260925 01:27:59.591032 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal/instance
I20260925 01:27:59.591465 7090 external_mini_cluster.cc:1428] Setting key 604419b3135e30d9594270582ea3bd9b
I20260925 01:27:59.594585 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.130:0
--local_ip_for_outbound_sockets=127.6.236.130
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--builtin_ntp_servers=127.6.236.148:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:27:59.596119 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:59.585921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.129:58731 (local address 127.6.236.190:37223)
0925 01:27:59.586367 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 01:27:59.586371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:27:59.587460 (+ 1089us) server_negotiation.cc:407] Connection header received
0925 01:27:59.589218 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:27:59.589224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:27:59.589327 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:27:59.589433 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:27:59.590556 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.591351 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.593033 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.593533 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.593637 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:59.595327 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:27:59.595334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:27:59.595339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:27:59.595360 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:27:59.595406 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:27:59.595421 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:27:59.595513 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:27:59.595698 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:27:59.595801 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":206,"threads_started":1}
I20260925 01:27:59.597033 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:27:59.586462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37223 (local address 127.6.236.129:58731)
0925 01:27:59.587307 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 01:27:59.587352 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:27:59.588954 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:27:59.589660 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:27:59.589668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:27:59.589764 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:27:59.590394 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:59.590414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.591498 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.591502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:59.592882 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:27:59.592895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:27:59.593787 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:27:59.593795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:27:59.594218 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:27:59.595008 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:27:59.595010 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:27:59.595050 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:27:59.595134 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:27:59.595508 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:27:59.595514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:27:59.595519 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:27:59.595646 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 01:27:59.595951 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":247,"threads_started":1}
I20260925 01:27:59.598536 27128 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37223
I20260925 01:27:59.598825 27128 heartbeater.cc:464] Registering TS with master...
I20260925 01:27:59.599543 27128 heartbeater.cc:511] Master 127.6.236.190:37223 requested a full tablet report, sending...
I20260925 01:27:59.601667 26942 ts_manager.cc:194] Registered new tserver with Master: fe861122cc644c7ca4aaa4e4b8b032c3 (127.6.236.129:45913)
I20260925 01:27:59.603183 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.129:58731
W20260925 01:27:59.913156 27132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:27:59.913419 27132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:27:59.913542 27132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:27:59.921118 27132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:27:59.921221 27132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:27:59.921360 27132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.130
I20260925 01:27:59.930382 27132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.236.130
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.130
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:27:59.931595 27132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:27:59.933359 27132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:27:59.945628 27138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.945672 27140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:27:59.945691 27137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:27:59.946635 27132 server_base.cc:1034] running on GCE node
I20260925 01:27:59.947253 27132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:27:59.948120 27132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:27:59.949331 27132 hybrid_clock.cc:648] HybridClock initialized: now 1790299679949275 us; error 45 us; skew 500 ppm
I20260925 01:27:59.949617 27132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:27:59.951663 27132 webserver.cc:460] Webserver started at http://127.6.236.130:36257/ using document root <none> and password file <none>
I20260925 01:27:59.952296 27132 fs_manager.cc:359] Metadata directory not provided
I20260925 01:27:59.952390 27132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:27:59.952692 27132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:27:59.955438 27132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data/instance:
uuid: "1258e4f1f0e04b7d837aafd2a0afd816"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "56c692945f3cf708db03e85e9d936eab"
server_key_iv: "a05d67aae93d5582788c555bee1b090e"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.956187 27132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance:
uuid: "1258e4f1f0e04b7d837aafd2a0afd816"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "56c692945f3cf708db03e85e9d936eab"
server_key_iv: "a05d67aae93d5582788c555bee1b090e"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.960954 27132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:27:59.964263 27146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:27:59.965476 27132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:27:59.965620 27132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal
uuid: "1258e4f1f0e04b7d837aafd2a0afd816"
format_stamp: "Formatted at 2026-09-25 01:27:59 on dist-test-slave-s0b2"
server_key: "56c692945f3cf708db03e85e9d936eab"
server_key_iv: "a05d67aae93d5582788c555bee1b090e"
server_key_version: "encryptionkey@0"
I20260925 01:27:59.965807 27132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:27:59.997437 27132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:27:59.998322 27132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:27:59.998595 27132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:27:59.999531 27132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:28:00.000993 27132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:28:00.001097 27132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:00.001197 27132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:28:00.001279 27132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:00.030028 27132 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.130:37307
I20260925 01:28:00.030089 27258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.130:37307 every 8 connection(s)
I20260925 01:28:00.031338 27132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/data/info.pb
I20260925 01:28:00.038695 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27132
I20260925 01:28:00.038874 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal/instance
I20260925 01:28:00.039294 7090 external_mini_cluster.cc:1428] Setting key 7cecb8be7516dd22f129c274b7b94481
I20260925 01:28:00.042043 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.236.131:0
--local_ip_for_outbound_sockets=127.6.236.131
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--builtin_ntp_servers=127.6.236.148:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:28:00.044306 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.033886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.130:38543 (local address 127.6.236.190:37223)
0925 01:28:00.034080 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.034085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.035623 (+ 1538us) server_negotiation.cc:407] Connection header received
0925 01:28:00.037298 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.037302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.037368 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.037450 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.038798 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.039542 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.041420 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.041830 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.041918 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.043488 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.043495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.043500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.043522 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.043559 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.043573 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.043653 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.043907 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.044002 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:28:00.045254 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.034535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37223 (local address 127.6.236.130:38543)
0925 01:28:00.035472 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.035517 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.037080 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.037745 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.037754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.037848 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:00.038627 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.038657 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.039697 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.039702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.041284 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.041299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.042046 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.042054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.042375 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.043137 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:28:00.043141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:00.043180 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:28:00.043277 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:00.043679 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:00.043693 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:00.043701 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.043862 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.044245 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":258,"threads_started":1}
I20260925 01:28:00.046576 27259 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37223
I20260925 01:28:00.046898 27259 heartbeater.cc:464] Registering TS with master...
I20260925 01:28:00.047740 27259 heartbeater.cc:511] Master 127.6.236.190:37223 requested a full tablet report, sending...
I20260925 01:28:00.049185 26942 ts_manager.cc:194] Registered new tserver with Master: 1258e4f1f0e04b7d837aafd2a0afd816 (127.6.236.130:37307)
I20260925 01:28:00.050153 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.130:38543
W20260925 01:28:00.362794 27262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:00.363062 27262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:00.363157 27262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:00.370278 27262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:00.370388 27262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:00.370503 27262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.236.131
I20260925 01:28:00.379487 27262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.236.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.236.131
--webserver_port=0
--tserver_master_addrs=127.6.236.190:37223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.236.131
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:00.380734 27262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:00.382575 27262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:00.395150 27268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:00.395220 27267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:00.395195 27270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:00.396178 27262 server_base.cc:1034] running on GCE node
I20260925 01:28:00.396737 27262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:00.397634 27262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:00.398835 27262 hybrid_clock.cc:648] HybridClock initialized: now 1790299680398782 us; error 44 us; skew 500 ppm
I20260925 01:28:00.399145 27262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:28:00.401501 27262 webserver.cc:460] Webserver started at http://127.6.236.131:43833/ using document root <none> and password file <none>
I20260925 01:28:00.402179 27262 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:00.402300 27262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:00.402581 27262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:00.405419 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data/instance:
uuid: "369eded9ecb04139996717746e118057"
format_stamp: "Formatted at 2026-09-25 01:28:00 on dist-test-slave-s0b2"
server_key: "e17a64d585d473c07ba91424bdff28fb"
server_key_iv: "09f96933498a06fbd00167ba46f1d759"
server_key_version: "encryptionkey@0"
I20260925 01:28:00.406126 27262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance:
uuid: "369eded9ecb04139996717746e118057"
format_stamp: "Formatted at 2026-09-25 01:28:00 on dist-test-slave-s0b2"
server_key: "e17a64d585d473c07ba91424bdff28fb"
server_key_iv: "09f96933498a06fbd00167ba46f1d759"
server_key_version: "encryptionkey@0"
I20260925 01:28:00.411015 27262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 01:28:00.414157 27276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:00.415427 27262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:28:00.415567 27262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal
uuid: "369eded9ecb04139996717746e118057"
format_stamp: "Formatted at 2026-09-25 01:28:00 on dist-test-slave-s0b2"
server_key: "e17a64d585d473c07ba91424bdff28fb"
server_key_iv: "09f96933498a06fbd00167ba46f1d759"
server_key_version: "encryptionkey@0"
I20260925 01:28:00.415781 27262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:00.430739 27262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:00.431504 27262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:00.431769 27262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:00.432626 27262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:28:00.434005 27262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:28:00.434084 27262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:00.434185 27262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:28:00.434263 27262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:00.462713 27262 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.131:37009
I20260925 01:28:00.462752 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.131:37009 every 8 connection(s)
I20260925 01:28:00.464003 27262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/data/info.pb
I20260925 01:28:00.465952 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27262
I20260925 01:28:00.466101 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal/instance
I20260925 01:28:00.466609 7090 external_mini_cluster.cc:1428] Setting key cb504effaffe59ea51833e0e97d502d1
I20260925 01:28:00.476112 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.466330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:47433 (local address 127.6.236.190:37223)
0925 01:28:00.466555 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.466562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.468014 (+ 1452us) server_negotiation.cc:407] Connection header received
0925 01:28:00.470143 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.470148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.470230 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.470347 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.471370 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.472089 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.473570 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.473978 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.474086 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.475354 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.475359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.475363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.475380 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.475411 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.475422 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.475493 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.475707 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.475801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:28:00.476982 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.466915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37223 (local address 127.6.236.131:47433)
0925 01:28:00.467877 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.467922 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.469902 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.470576 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.470584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.470667 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:00.471211 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.471228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.472236 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.472240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.473438 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.473450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.474178 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.474185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.474444 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.475055 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:28:00.475058 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:00.475095 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:28:00.475167 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:00.475503 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:00.475514 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:00.475522 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.475666 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.476033 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":275,"threads_started":1}
I20260925 01:28:00.478097 27389 heartbeater.cc:347] Connected to a master server at 127.6.236.190:37223
I20260925 01:28:00.478399 27389 heartbeater.cc:464] Registering TS with master...
I20260925 01:28:00.479075 27389 heartbeater.cc:511] Master 127.6.236.190:37223 requested a full tablet report, sending...
I20260925 01:28:00.480470 26942 ts_manager.cc:194] Registered new tserver with Master: 369eded9ecb04139996717746e118057 (127.6.236.131:37009)
I20260925 01:28:00.481479 26942 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.236.131:47433
I20260925 01:28:00.481667 7090 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:28:00.490765 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.485477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.6.236.190:37223)
0925 01:28:00.485628 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.485634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.486004 (+ 370us) server_negotiation.cc:407] Connection header received
0925 01:28:00.486155 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.486160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.486222 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.486328 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.486785 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.487441 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.488680 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.489065 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.489167 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.490106 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.490112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.490116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.490133 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.490163 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.490174 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.490267 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.490404 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.490489 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:28:00.494154 26942 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:28:00.501137 26942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:28:00.527935 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.512301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.131:37009 (local address 127.0.0.1:52484)
0925 01:28:00.513113 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.513158 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.513328 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.518852 (+ 5524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.518859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.518890 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:28:00.519285 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.519301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.521210 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.521231 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.523868 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.523884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.525866 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.525877 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.526112 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.526168 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.526304 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.527512 (+ 1208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":80,"thread_start_us":5921,"threads_started":1}
I20260925 01:28:00.528849 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.512467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54188)
0925 01:28:00.518571 (+ 6104us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.518588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.518707 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.519688 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.519693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.519717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:28:00.520058 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.520069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.521912 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.521918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.525807 (+ 3889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.525823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.528093 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.528098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.528271 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.528307 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.528397 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.528531 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5926,"thread_start_us":463,"threads_started":1}
I20260925 01:28:00.529460 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.512482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.6.236.131:37009)
0925 01:28:00.518171 (+ 5689us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.518179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.518213 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:28:00.518331 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.518339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.518584 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.518814 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:28:00.519470 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.521082 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.524075 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.525630 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.528473 (+ 2843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.528763 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.528836 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.529014 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5525,"thread_start_us":5398,"threads_started":1}
I20260925 01:28:00.530018 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.512976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.6.236.129:45913)
0925 01:28:00.514494 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.514504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.518822 (+ 4318us) server_negotiation.cc:407] Connection header received
0925 01:28:00.518924 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.518931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.519261 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.519438 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:28:00.520239 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.521752 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.526040 (+ 4288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.527571 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.527701 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.527962 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.529370 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.529551 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":1179,"threads_started":1}
I20260925 01:28:00.534073 27063 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 (DEFAULT_TABLE table=test-table [id=49706d8f35624b38970895c797a4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:28:00.535385 27063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd575fad74a40b89e2e79301d1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:00.535079 27324 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 (DEFAULT_TABLE table=test-table [id=49706d8f35624b38970895c797a4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:28:00.536489 27324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd575fad74a40b89e2e79301d1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:00.539809 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.519182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37307 (local address 127.0.0.1:46038)
0925 01:28:00.528179 (+ 8997us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.528210 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.528313 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.533505 (+ 5192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.533510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.533537 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:28:00.533686 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.533706 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.535576 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.535581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.537329 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.537344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.539021 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.539028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.539210 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.539244 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.539339 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.539473 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8895,"thread_start_us":12212,"threads_started":1}
I20260925 01:28:00.539898 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.518974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.6.236.130:37307)
0925 01:28:00.532914 (+ 13940us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.532921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.532951 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:28:00.533055 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.533062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.533305 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.533501 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:28:00.533839 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.535400 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.537502 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.538852 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.538982 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.539237 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.539393 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.539547 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13774,"thread_start_us":13664,"threads_started":1}
I20260925 01:28:00.543825 27194 tablet_service.cc:1467] Processing CreateTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 (DEFAULT_TABLE table=test-table [id=49706d8f35624b38970895c797a4dfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:28:00.545235 27194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1cd575fad74a40b89e2e79301d1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:00.548091 27407 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Bootstrap starting.
I20260925 01:28:00.550127 27408 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Bootstrap starting.
I20260925 01:28:00.552305 27407 tablet_bootstrap.cc:654] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:00.553479 27408 tablet_bootstrap.cc:654] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:00.553544 27407 log.cc:824] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:00.554706 27408 log.cc:824] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:00.556640 27407 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: No bootstrap required, opened a new log
I20260925 01:28:00.556910 27407 ts_tablet_manager.cc:1397] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 01:28:00.557338 27411 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Bootstrap starting.
I20260925 01:28:00.559180 27408 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: No bootstrap required, opened a new log
I20260925 01:28:00.559442 27408 ts_tablet_manager.cc:1397] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260925 01:28:00.560745 27411 tablet_bootstrap.cc:654] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:00.561620 27411 log.cc:824] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:00.563844 27411 tablet_bootstrap.cc:492] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: No bootstrap required, opened a new log
I20260925 01:28:00.564325 27411 ts_tablet_manager.cc:1397] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 01:28:00.566911 27407 raft_consensus.cc:348] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.567338 27407 raft_consensus.cc:374] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:00.567437 27407 raft_consensus.cc:729] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe861122cc644c7ca4aaa4e4b8b032c3, State: Initialized, Role: FOLLOWER
I20260925 01:28:00.568053 27407 consensus_queue.cc:260] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.569278 27128 heartbeater.cc:503] Master 127.6.236.190:37223 was elected leader, sending a full tablet report...
I20260925 01:28:00.569264 27408 raft_consensus.cc:348] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.569670 27408 raft_consensus.cc:374] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:00.569759 27408 raft_consensus.cc:729] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369eded9ecb04139996717746e118057, State: Initialized, Role: FOLLOWER
I20260925 01:28:00.569983 27407 ts_tablet_manager.cc:1428] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 01:28:00.570492 27408 consensus_queue.cc:260] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.571498 27389 heartbeater.cc:503] Master 127.6.236.190:37223 was elected leader, sending a full tablet report...
I20260925 01:28:00.572773 27414 raft_consensus.cc:482] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:28:00.572882 27411 raft_consensus.cc:348] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.573293 27411 raft_consensus.cc:374] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:00.573064 27414 raft_consensus.cc:504] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.573395 27411 raft_consensus.cc:729] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1258e4f1f0e04b7d837aafd2a0afd816, State: Initialized, Role: FOLLOWER
I20260925 01:28:00.573974 27411 consensus_queue.cc:260] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.576860 27414 leader_election.cc:288] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe861122cc644c7ca4aaa4e4b8b032c3 (127.6.236.129:45913), 1258e4f1f0e04b7d837aafd2a0afd816 (127.6.236.130:37307)
I20260925 01:28:00.580684 27408 ts_tablet_manager.cc:1428] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260925 01:28:00.583429 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.575817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.130:37307 (local address 127.6.236.131:45309)
0925 01:28:00.576017 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.576039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.576167 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.576711 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.576716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.576742 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:28:00.576945 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.576955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.578540 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.578547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.580670 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.580687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.582623 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.582629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.582809 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.582886 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.582969 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.583105 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:28:00.583455 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.576200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:45309 (local address 127.6.236.130:37307)
0925 01:28:00.576394 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.576400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.576422 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:28:00.576490 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.576495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.576568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.576659 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:28:00.577155 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.578380 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.580924 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.582184 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.582462 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.582530 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.583008 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.583202 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:28:00.584754 27214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd575fad74a40b89e2e79301d1ce2" candidate_uuid: "369eded9ecb04139996717746e118057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" is_pre_election: true
I20260925 01:28:00.585325 27214 raft_consensus.cc:2454] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369eded9ecb04139996717746e118057 in term 0.
W20260925 01:28:00.585969 27129 tablet.cc:2369] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:28:00.587172 27279 leader_election.cc:302] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1258e4f1f0e04b7d837aafd2a0afd816, 369eded9ecb04139996717746e118057; no voters:
I20260925 01:28:00.588303 27415 raft_consensus.cc:2781] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:28:00.588476 27415 raft_consensus.cc:482] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:28:00.588631 27415 raft_consensus.cc:3037] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:00.592665 27415 raft_consensus.cc:504] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.594094 27259 heartbeater.cc:503] Master 127.6.236.190:37223 was elected leader, sending a full tablet report...
I20260925 01:28:00.594856 27411 ts_tablet_manager.cc:1428] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Time spent starting tablet: real 0.030s user 0.011s sys 0.016s
I20260925 01:28:00.595005 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.587764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.6.236.131:60793)
0925 01:28:00.587888 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.587904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:00.588007 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:00.588471 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:00.588476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:00.588502 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:28:00.588685 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.588708 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.590005 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.590009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.591858 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:00.591872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.594281 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.594288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:00.594498 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.594530 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:28:00.594619 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:28:00.594743 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 01:28:00.595721 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.587687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.236.131:60793 (local address 127.6.236.129:45913)
0925 01:28:00.587882 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.587888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.587952 (+ 64us) server_negotiation.cc:407] Connection header received
0925 01:28:00.588123 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.588128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.588222 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.588306 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:28:00.588857 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.589855 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.592044 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.593320 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.593466 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.593534 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.595357 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.595480 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:28:00.596369 27415 leader_election.cc:288] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [CANDIDATE]: Term 1 election: Requested vote from peers fe861122cc644c7ca4aaa4e4b8b032c3 (127.6.236.129:45913), 1258e4f1f0e04b7d837aafd2a0afd816 (127.6.236.130:37307)
I20260925 01:28:00.596975 27083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd575fad74a40b89e2e79301d1ce2" candidate_uuid: "369eded9ecb04139996717746e118057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" is_pre_election: true
I20260925 01:28:00.597262 27214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd575fad74a40b89e2e79301d1ce2" candidate_uuid: "369eded9ecb04139996717746e118057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1258e4f1f0e04b7d837aafd2a0afd816"
I20260925 01:28:00.597433 27214 raft_consensus.cc:3037] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:00.597546 27083 raft_consensus.cc:2454] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 369eded9ecb04139996717746e118057 in term 0.
I20260925 01:28:00.597018 27082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1cd575fad74a40b89e2e79301d1ce2" candidate_uuid: "369eded9ecb04139996717746e118057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3"
I20260925 01:28:00.598094 27082 raft_consensus.cc:3037] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:00.600289 27214 raft_consensus.cc:2454] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 369eded9ecb04139996717746e118057 in term 1.
I20260925 01:28:00.600636 27279 leader_election.cc:302] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1258e4f1f0e04b7d837aafd2a0afd816, 369eded9ecb04139996717746e118057; no voters:
I20260925 01:28:00.600875 27082 raft_consensus.cc:2454] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 369eded9ecb04139996717746e118057 in term 1.
I20260925 01:28:00.600883 27415 raft_consensus.cc:2781] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:00.601038 27415 raft_consensus.cc:686] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 1 LEADER]: Becoming Leader. State: Replica: 369eded9ecb04139996717746e118057, State: Running, Role: LEADER
I20260925 01:28:00.601565 27415 consensus_queue.cc:237] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } }
I20260925 01:28:00.606034 26941 catalog_manager.cc:5495] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 reported cstate change: term changed from 0 to 1, leader changed from <none> to 369eded9ecb04139996717746e118057 (127.6.236.131). New cstate: current_term: 1 leader_uuid: "369eded9ecb04139996717746e118057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "369eded9ecb04139996717746e118057" member_type: VOTER last_known_addr { host: "127.6.236.131" port: 37009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 } health_report { overall_health: UNKNOWN } } }
I20260925 01:28:00.625378 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.620845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.6.236.131:37009)
0925 01:28:00.621020 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.621025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.621042 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:28:00.621172 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.621175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.621257 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.621373 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:00.621646 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.622387 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.623620 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.624002 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.624086 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.624461 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:00.624943 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:00.625045 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.625083 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.625142 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:28:00.632841 27214 raft_consensus.cc:1261] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 1 FOLLOWER]: Refusing update from remote peer 369eded9ecb04139996717746e118057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:00.632836 27082 raft_consensus.cc:1261] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 1 FOLLOWER]: Refusing update from remote peer 369eded9ecb04139996717746e118057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:00.633675 27414 consensus_queue.cc:1035] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe861122cc644c7ca4aaa4e4b8b032c3" member_type: VOTER last_known_addr { host: "127.6.236.129" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:28:00.633965 27415 consensus_queue.cc:1035] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1258e4f1f0e04b7d837aafd2a0afd816" member_type: VOTER last_known_addr { host: "127.6.236.130" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:28:00.646663 27419 mvcc.cc:205] Tried to move back new op lower bound from 7333067491863658496 to 7333067491748003840. Current Snapshot: MvccSnapshot[applied={T|T < 7333067491863658496}]
I20260925 01:28:00.658535 26941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:28:00.658787 26941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:28:00.662207 26941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d61c817804149618142ec6c3f62aa50: Sending DeleteTablet for 3 replicas of tablet ae1cd575fad74a40b89e2e79301d1ce2
I20260925 01:28:00.663295 27324 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:28:00 UTC) from {username='slave'} at 127.0.0.1:52484
I20260925 01:28:00.663365 27063 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:28:00 UTC) from {username='slave'} at 127.0.0.1:54188
I20260925 01:28:00.663540 27194 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1cd575fad74a40b89e2e79301d1ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:28:00 UTC) from {username='slave'} at 127.0.0.1:46038
I20260925 01:28:00.664594 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27001
I20260925 01:28:00.675925 27431 tablet_replica.cc:330] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: stopping tablet replica
I20260925 01:28:00.676431 27431 raft_consensus.cc:2229] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:28:00.676676 27430 tablet_replica.cc:330] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: stopping tablet replica
I20260925 01:28:00.676997 27431 raft_consensus.cc:2258] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:28:00.677224 27430 raft_consensus.cc:2229] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:28:00.677739 27430 raft_consensus.cc:2258] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:28:00.677871 27429 tablet_replica.cc:330] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: stopping tablet replica
I20260925 01:28:00.678475 27429 raft_consensus.cc:2229] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:28:00.679106 27429 raft_consensus.cc:2258] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:28:00.679265 27431 ts_tablet_manager.cc:1905] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:28:00.681049 27430 ts_tablet_manager.cc:1905] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:28:00.684048 27429 ts_tablet_manager.cc:1905] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:28:00.686038 27431 ts_tablet_manager.cc:1918] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:28:00.686188 27431 log.cc:1196] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-1/wal/wals/ae1cd575fad74a40b89e2e79301d1ce2
I20260925 01:28:00.686743 27431 ts_tablet_manager.cc:1939] T ae1cd575fad74a40b89e2e79301d1ce2 P 1258e4f1f0e04b7d837aafd2a0afd816: Deleting consensus metadata
I20260925 01:28:00.688812 26928 catalog_manager.cc:4841] TS 1258e4f1f0e04b7d837aafd2a0afd816 (127.6.236.130:37307): tablet ae1cd575fad74a40b89e2e79301d1ce2 (table test-table [id=49706d8f35624b38970895c797a4dfe5]) successfully deleted
I20260925 01:28:00.691126 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.666072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.6.236.129:45913)
0925 01:28:00.666264 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.666271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.681131 (+ 14860us) server_negotiation.cc:407] Connection header received
0925 01:28:00.681622 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.681628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.681729 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.681806 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.682464 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.683444 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.685416 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.686001 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.686097 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.689406 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.689417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.689433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.690179 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.690269 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.690295 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.690405 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.690606 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.690733 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:28:00.691108 27429 ts_tablet_manager.cc:1918] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:28:00.691427 27429 log.cc:1196] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-2/wal/wals/ae1cd575fad74a40b89e2e79301d1ce2
I20260925 01:28:00.692117 27429 ts_tablet_manager.cc:1939] T ae1cd575fad74a40b89e2e79301d1ce2 P 369eded9ecb04139996717746e118057: Deleting consensus metadata
I20260925 01:28:00.692274 27123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:00.693806 26928 catalog_manager.cc:4841] TS 369eded9ecb04139996717746e118057 (127.6.236.131:37009): tablet ae1cd575fad74a40b89e2e79301d1ce2 (table test-table [id=49706d8f35624b38970895c797a4dfe5]) successfully deleted
I20260925 01:28:00.800348 27430 ts_tablet_manager.cc:1918] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:28:00.800841 27430 log.cc:1196] T ae1cd575fad74a40b89e2e79301d1ce2 P fe861122cc644c7ca4aaa4e4b8b032c3: Deleting WAL directory at /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299375376939-7090-0/minicluster-data/ts-0/wal/wals/ae1cd575fad74a40b89e2e79301d1ce2
I20260925 01:28:00.801334 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27001
I20260925 01:28:00.822124 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27132
W20260925 01:28:00.822111 26929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.236.129:45913 (error 108)
I20260925 01:28:00.824468 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.823975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54196)
0925 01:28:00.824160 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.824281 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:28:00.824733 26929 catalog_manager.cc:4584] TS fe861122cc644c7ca4aaa4e4b8b032c3 (127.6.236.129:45913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae1cd575fad74a40b89e2e79301d1ce2: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
I20260925 01:28:00.829919 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.823265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.6.236.130:37307)
0925 01:28:00.823437 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.823442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.823459 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:28:00.823519 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.823523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.823612 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.823706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.824175 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.825137 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.827096 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.827481 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.827574 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.828624 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.828632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.828647 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.829200 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.829248 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.829287 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.829391 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.829519 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.829623 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:28:00.830535 27254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:00.878659 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.878090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54200)
0925 01:28:00.878377 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.878482 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 01:28:00.879828 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.879407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54210)
0925 01:28:00.879561 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.879664 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 01:28:00.916672 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27132
I20260925 01:28:00.931713 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27262
I20260925 01:28:00.939136 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.932445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.6.236.131:37009)
0925 01:28:00.932612 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:00.932617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:00.932682 (+ 65us) server_negotiation.cc:407] Connection header received
0925 01:28:00.932880 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:00.932884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:00.932962 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:00.933044 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:00.933548 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.934386 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.936107 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:00.936515 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:00.936606 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:00.937714 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:00.937723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:00.937737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:00.938394 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:00.938446 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:00.938466 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:00.938550 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:00.938715 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 01:28:00.938806 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:28:00.939893 27384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:00.940227 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.939801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54226)
0925 01:28:00.939977 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.940072 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:28:00.941532 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:00.941075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54232)
0925 01:28:00.941295 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:00.941377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 01:28:01.022071 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:01.021584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54242)
0925 01:28:01.021819 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:01.021918 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:28:01.023185 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:01.022769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.129:45913 (local address 127.0.0.1:54248)
0925 01:28:01.022968 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:01.023048 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.129:45913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 01:28:01.056712 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27262
I20260925 01:28:01.072850 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 26911
I20260925 01:28:01.073912 26972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:01.183578 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 26911
2026-09-25T01:28:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2601 ms)
[----------] 6 tests from EncryptionPolicyTest (20588 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:01 dist-test-slave-s0b2 krb5kdc[27440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:01 dist-test-slave-s0b2 krb5kdc[27440](info): set up 2 sockets
Sep 25 01:28:01 dist-test-slave-s0b2 krb5kdc[27440](info): commencing operation
krb5kdc: starting...
W20260925 01:28:02.207281 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:02 dist-test-slave-s0b2 krb5kdc[27440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:02.448925 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:02.766489 27453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:02.766770 27453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:02.766870 27453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:02.774179 27453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:02.774313 27453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:02.774370 27453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:02.774430 27453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:02.774473 27453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:02.784266 27453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:02.785447 27453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:02.786715 27453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:02.797883 27458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:02.797855 27459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:02.797935 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:02.798642 27453 server_base.cc:1034] running on GCE node
I20260925 01:28:02.799372 27453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:02.800724 27453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:02.801929 27453 hybrid_clock.cc:648] HybridClock initialized: now 1790299682801876 us; error 42 us; skew 500 ppm
I20260925 01:28:02.802286 27453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:02 dist-test-slave-s0b2 krb5kdc[27440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:02.805186 27453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:02.806926 27453 webserver.cc:460] Webserver started at http://127.0.0.1:42443/ using document root <none> and password file <none>
I20260925 01:28:02.807773 27453 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:02.807907 27453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:02.808239 27453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:02.811334 27453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "0652894f2b304857844617e87521b416"
format_stamp: "Formatted at 2026-09-25 01:28:02 on dist-test-slave-s0b2"
server_key: "bbd8e4d8165319aebabfe232ab2831f4"
server_key_iv: "8469bf3a294b4fae30fa3f461dea74c5"
server_key_version: "encryptionkey@0"
I20260925 01:28:02.812011 27453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "0652894f2b304857844617e87521b416"
format_stamp: "Formatted at 2026-09-25 01:28:02 on dist-test-slave-s0b2"
server_key: "bbd8e4d8165319aebabfe232ab2831f4"
server_key_iv: "8469bf3a294b4fae30fa3f461dea74c5"
server_key_version: "encryptionkey@0"
I20260925 01:28:02.816808 27453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:28:02.820079 27468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:02.821334 27453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:28:02.821465 27453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "0652894f2b304857844617e87521b416"
format_stamp: "Formatted at 2026-09-25 01:28:02 on dist-test-slave-s0b2"
server_key: "bbd8e4d8165319aebabfe232ab2831f4"
server_key_iv: "8469bf3a294b4fae30fa3f461dea74c5"
server_key_version: "encryptionkey@0"
I20260925 01:28:02.821638 27453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:02.835484 27453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:02.839062 27453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:02.839294 27453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:02.855341 27453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40273
I20260925 01:28:02.855362 27519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40273 every 8 connection(s)
I20260925 01:28:02.856649 27453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:02.860225 27520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:02.861986 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27453
I20260925 01:28:02.862182 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:02.862812 7090 external_mini_cluster.cc:1428] Setting key 91f2cef23c7933849095c81881021bde
I20260925 01:28:02.871603 27520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416: Bootstrap starting.
I20260925 01:28:02.875389 27520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:02.876497 27520 log.cc:824] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:28:02 dist-test-slave-s0b2 krb5kdc[27440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:28:02.879426 27520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416: No bootstrap required, opened a new log
I20260925 01:28:02.884771 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:02.864618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51381 (local address 127.0.0.1:40273)
0925 01:28:02.865373 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:02.865385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:02.865426 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:28:02.867097 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:02.867119 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:02.867505 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:02.867823 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:02.869521 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:02.870882 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:02.872597 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:02.873257 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:02.873400 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:02.877709 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:02.877717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:02.877732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:02.877772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:02.880684 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:02.881449 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:02.881455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:02.881462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:02.881654 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:02.882093 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:02.882097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:02.882099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:02.882665 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:02.882965 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:02.883466 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 01:28:02.883769 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260925 01:28:02.887063 27520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 01:28:02.887416 27520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:02.887513 27520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0652894f2b304857844617e87521b416, State: Initialized, Role: FOLLOWER
I20260925 01:28:02.888110 27520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 01:28:02.888401 27520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:02.888531 27520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:02.888543 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:02.888672 27520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:02.890818 27520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 01:28:02.891350 27520 leader_election.cc:302] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0652894f2b304857844617e87521b416; no voters:
I20260925 01:28:02.891956 27520 leader_election.cc:288] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:02.892077 27528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:02.892660 27528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [term 1 LEADER]: Becoming Leader. State: Replica: 0652894f2b304857844617e87521b416, State: Running, Role: LEADER
I20260925 01:28:02.893251 27528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 01:28:02.894582 27520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:02.896389 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0652894f2b304857844617e87521b416. Latest consensus state: current_term: 1 leader_uuid: "0652894f2b304857844617e87521b416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260925 01:28:02.896768 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:02.897291 27531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0652894f2b304857844617e87521b416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652894f2b304857844617e87521b416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260925 01:28:02.897570 27531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:02.900622 27535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:02.907387 27535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:02.912315 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:02.893386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50981 (local address 127.0.0.1:40273)
0925 01:28:02.893534 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:02.893540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:02.893964 (+ 424us) server_negotiation.cc:407] Connection header received
0925 01:28:02.894216 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:02.894221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:02.894346 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:02.894415 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:02.896435 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:02.897618 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:02.899485 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:02.900175 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:02.900271 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:02.903593 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:02.903601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:02.903606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:02.903648 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:02.907808 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:02.908856 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:02.908862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:02.908866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:02.909017 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:02.909585 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:02.909591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:02.909594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:02.910058 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:02.911547 (+ 1489us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:02.911741 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 01:28:02.911932 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 01:28:02.917603 27535 catalog_manager.cc:1347] Generated new cluster ID: b0ea91d12155486d841eeb6af57d1b2b
I20260925 01:28:02.917732 27535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:02.935837 27535 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:02.937598 27535 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:02.945859 27535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0652894f2b304857844617e87521b416: Generated new TSK 0
I20260925 01:28:02.946851 27535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:02.953233 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27453
I20260925 01:28:02.954804 27515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:03.043404 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27453
W20260925 01:28:03.057920 27438 connection.cc:535] client connection to 127.0.0.1:40273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40273): Connection reset by peer (error 104)
2026-09-25T01:28:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:03 dist-test-slave-s0b2 krb5kdc[27553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:03 dist-test-slave-s0b2 krb5kdc[27553](info): set up 2 sockets
Sep 25 01:28:03 dist-test-slave-s0b2 krb5kdc[27553](info): commencing operation
krb5kdc: starting...
W20260925 01:28:04.055261 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:04 dist-test-slave-s0b2 krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:04.291203 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:04.610481 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:04.610775 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:04.610905 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:04.618115 27566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:04.618220 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:04.618319 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:04.618402 27566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:04.618458 27566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:04.628028 27566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:04.629298 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:04.630632 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:04.641448 27571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:04.642282 27572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:04.642309 27574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:04.642400 27566 server_base.cc:1034] running on GCE node
I20260925 01:28:04.643237 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:04.644550 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:04.645756 27566 hybrid_clock.cc:648] HybridClock initialized: now 1790299684645715 us; error 31 us; skew 500 ppm
I20260925 01:28:04.646072 27566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:04 dist-test-slave-s0b2 krb5kdc[27553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:04.648975 27566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:04.650447 27566 webserver.cc:460] Webserver started at http://127.0.0.1:37555/ using document root <none> and password file <none>
I20260925 01:28:04.651121 27566 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:04.651230 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:04.651520 27566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:04.654508 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "4e60a9440eaf4d2b827f87e58d85d418"
format_stamp: "Formatted at 2026-09-25 01:28:04 on dist-test-slave-s0b2"
server_key: "413ccbf8c70d209670df8f9b0d065945"
server_key_iv: "4dc1687c5a66f7d6e6de5d22f1ff47f8"
server_key_version: "encryptionkey@0"
I20260925 01:28:04.655242 27566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "4e60a9440eaf4d2b827f87e58d85d418"
format_stamp: "Formatted at 2026-09-25 01:28:04 on dist-test-slave-s0b2"
server_key: "413ccbf8c70d209670df8f9b0d065945"
server_key_iv: "4dc1687c5a66f7d6e6de5d22f1ff47f8"
server_key_version: "encryptionkey@0"
I20260925 01:28:04.660123 27566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:28:04.663408 27581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:04.664710 27566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:28:04.664866 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "4e60a9440eaf4d2b827f87e58d85d418"
format_stamp: "Formatted at 2026-09-25 01:28:04 on dist-test-slave-s0b2"
server_key: "413ccbf8c70d209670df8f9b0d065945"
server_key_iv: "4dc1687c5a66f7d6e6de5d22f1ff47f8"
server_key_version: "encryptionkey@0"
I20260925 01:28:04.665035 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:04.682013 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:04.685849 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:04.686118 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:04.703454 27566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42993
I20260925 01:28:04.703475 27632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42993 every 8 connection(s)
I20260925 01:28:04.704866 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:04.708590 27633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:04.712729 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27566
I20260925 01:28:04.712908 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:04.713352 7090 external_mini_cluster.cc:1428] Setting key 6b16e1d2ed270abc5af5a5b1272c736f
I20260925 01:28:04.719173 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418: Bootstrap starting.
I20260925 01:28:04.722128 27633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:04.723158 27633 log.cc:824] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:04.726064 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:04.715123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55743 (local address 127.0.0.1:42993)
0925 01:28:04.715885 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:04.715897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:04.715941 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:28:04.717926 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:04.717951 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:04.718384 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:04.718733 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:04.719272 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:04.720593 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:04.721902 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:04.722644 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:04.722823 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:04.723938 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:04.723946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:04.723959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:04.724498 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:04.724577 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:04.724601 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:04.724731 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:04.725009 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 01:28:04.725161 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":329,"threads_started":1}
I20260925 01:28:04.726110 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418: No bootstrap required, opened a new log
I20260925 01:28:04.729687 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:04.733654 27633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260925 01:28:04.734109 27633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:04.734217 27633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e60a9440eaf4d2b827f87e58d85d418, State: Initialized, Role: FOLLOWER
I20260925 01:28:04.734906 27633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260925 01:28:04.735137 27633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:04.735301 27633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:04.735477 27633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:04.738077 27633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260925 01:28:04.738646 27633 leader_election.cc:302] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e60a9440eaf4d2b827f87e58d85d418; no voters:
I20260925 01:28:04.739244 27633 leader_election.cc:288] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:04.739389 27643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:04.740669 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:04.733788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40525 (local address 127.0.0.1:42993)
0925 01:28:04.733963 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:04.733974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:04.734291 (+ 317us) server_negotiation.cc:407] Connection header received
0925 01:28:04.734499 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:04.734505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:04.734588 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:04.734695 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:04.735143 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:04.736250 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:04.737537 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:04.738173 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:04.738333 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:04.739589 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:04.739597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:04.739610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:04.739637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:04.739687 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:04.739715 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:04.740235 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:04.740310 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 01:28:04.740359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 01:28:04.741392 27633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:04.742523 27643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [term 1 LEADER]: Becoming Leader. State: Replica: 4e60a9440eaf4d2b827f87e58d85d418, State: Running, Role: LEADER
I20260925 01:28:04.743023 27643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } }
I20260925 01:28:04.745520 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e60a9440eaf4d2b827f87e58d85d418. Latest consensus state: current_term: 1 leader_uuid: "4e60a9440eaf4d2b827f87e58d85d418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } } }
I20260925 01:28:04.745976 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:04.745590 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e60a9440eaf4d2b827f87e58d85d418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e60a9440eaf4d2b827f87e58d85d418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42993 } } }
I20260925 01:28:04.746562 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:04.751228 27651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:04.755959 27651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:04.764271 27651 catalog_manager.cc:1347] Generated new cluster ID: 5ce067d2dce14ac584db16c9ddb74a35
I20260925 01:28:04.764374 27651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:04.781764 27651 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:04.783658 27651 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:04.793831 27651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e60a9440eaf4d2b827f87e58d85d418: Generated new TSK 0
I20260925 01:28:04.794816 27651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:04.811081 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27566
W20260925 01:28:04.812235 27628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55743
W20260925 01:28:04.813045 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:04.813225 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27566
W20260925 01:28:04.828455 27551 connection.cc:535] client connection to 127.0.0.1:42993 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42993): Connection reset by peer (error 104)
2026-09-25T01:28:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1770 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:04 dist-test-slave-s0b2 krb5kdc[27664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:04 dist-test-slave-s0b2 krb5kdc[27664](info): set up 2 sockets
Sep 25 01:28:04 dist-test-slave-s0b2 krb5kdc[27664](info): commencing operation
krb5kdc: starting...
W20260925 01:28:05.831203 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:05 dist-test-slave-s0b2 krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:06.067358 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:06.386152 27677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:06.386479 27677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:06.386580 27677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:06.394136 27677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:06.394275 27677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:06.394357 27677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:06.394420 27677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:06.394462 27677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:06.404410 27677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:06.405645 27677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:06.406958 27677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:06.418327 27682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:06.418368 27685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:06.418406 27683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:06.419111 27677 server_base.cc:1034] running on GCE node
I20260925 01:28:06.419878 27677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:06.421235 27677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:06.422456 27677 hybrid_clock.cc:648] HybridClock initialized: now 1790299686422403 us; error 45 us; skew 500 ppm
I20260925 01:28:06.422791 27677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:06 dist-test-slave-s0b2 krb5kdc[27664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:06.425691 27677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:06.427158 27677 webserver.cc:460] Webserver started at http://127.0.0.1:36037/ using document root <none> and password file <none>
I20260925 01:28:06.427834 27677 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:06.427948 27677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:06.428243 27677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:06.431267 27677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "ee8beacd98e84c1c85942c34be89b91a"
format_stamp: "Formatted at 2026-09-25 01:28:06 on dist-test-slave-s0b2"
server_key: "0f5a2dfb5a39a86ad37edf0da9b2f1de"
server_key_iv: "951c32639aec2de5892c21d92215880c"
server_key_version: "encryptionkey@0"
I20260925 01:28:06.432009 27677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "ee8beacd98e84c1c85942c34be89b91a"
format_stamp: "Formatted at 2026-09-25 01:28:06 on dist-test-slave-s0b2"
server_key: "0f5a2dfb5a39a86ad37edf0da9b2f1de"
server_key_iv: "951c32639aec2de5892c21d92215880c"
server_key_version: "encryptionkey@0"
I20260925 01:28:06.437063 27677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:28:06.440526 27692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:06.441838 27677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 01:28:06.441987 27677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "ee8beacd98e84c1c85942c34be89b91a"
format_stamp: "Formatted at 2026-09-25 01:28:06 on dist-test-slave-s0b2"
server_key: "0f5a2dfb5a39a86ad37edf0da9b2f1de"
server_key_iv: "951c32639aec2de5892c21d92215880c"
server_key_version: "encryptionkey@0"
I20260925 01:28:06.442160 27677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:06.447784 27677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:06.451673 27677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:06.451939 27677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:06.468672 27677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39323
I20260925 01:28:06.468724 27743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39323 every 8 connection(s)
I20260925 01:28:06.470018 27677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:06.470512 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27677
I20260925 01:28:06.470669 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:06.471244 7090 external_mini_cluster.cc:1428] Setting key 257007d170138240f954f5278398dbf4
I20260925 01:28:06.474295 27744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:06.478788 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:06.472965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57159 (local address 127.0.0.1:39323)
0925 01:28:06.473701 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:06.473715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:06.473755 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:28:06.475570 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:06.475592 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:06.475909 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:06.476250 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:06.477204 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:06.477210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:06.477223 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:06.477243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:06.477312 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:06.477335 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:06.477541 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:06.477676 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 01:28:06.477845 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":319,"threads_started":1}
I20260925 01:28:06.482605 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:06.485507 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a: Bootstrap starting.
I20260925 01:28:06.489111 27744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:06.489137 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:06.486805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36711 (local address 127.0.0.1:39323)
0925 01:28:06.487021 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:06.487027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:06.487330 (+ 303us) server_negotiation.cc:407] Connection header received
0925 01:28:06.487499 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:06.487503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:06.487549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:06.487638 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:06.488520 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:06.488524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:06.488527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:06.488541 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:06.488587 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:06.488602 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:06.488693 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:06.488818 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 01:28:06.488860 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 01:28:06.490396 27744 log.cc:824] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:06.492957 27744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a: No bootstrap required, opened a new log
I20260925 01:28:06.500059 27744 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260925 01:28:06.500434 27744 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:06.500524 27744 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8beacd98e84c1c85942c34be89b91a, State: Initialized, Role: FOLLOWER
I20260925 01:28:06.501113 27744 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260925 01:28:06.501332 27744 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:06.501488 27744 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:06.501681 27744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:06.503916 27744 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260925 01:28:06.504383 27744 leader_election.cc:302] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8beacd98e84c1c85942c34be89b91a; no voters:
I20260925 01:28:06.504936 27744 leader_election.cc:288] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:06.505056 27754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:06.505666 27754 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [term 1 LEADER]: Becoming Leader. State: Replica: ee8beacd98e84c1c85942c34be89b91a, State: Running, Role: LEADER
I20260925 01:28:06.506170 27754 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260925 01:28:06.506940 27744 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:06.508631 27755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8beacd98e84c1c85942c34be89b91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } } }
I20260925 01:28:06.509019 27755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:06.509301 27756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8beacd98e84c1c85942c34be89b91a. Latest consensus state: current_term: 1 leader_uuid: "ee8beacd98e84c1c85942c34be89b91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8beacd98e84c1c85942c34be89b91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } } }
I20260925 01:28:06.509588 27756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:06.511112 27759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:06.515995 27759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:06.524379 27759 catalog_manager.cc:1347] Generated new cluster ID: a2e823927dbf4d0d944e6077eb0202ae
I20260925 01:28:06.524490 27759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:06.536921 27759 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:06.537431 27759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:06.557322 27759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8beacd98e84c1c85942c34be89b91a: Generated new TSK 0
I20260925 01:28:06.558516 27759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:06.575567 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27677
W20260925 01:28:06.576653 27739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57159
W20260925 01:28:06.577383 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:06.577502 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27677
2026-09-25T01:28:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1762 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:06 dist-test-slave-s0b2 krb5kdc[27775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:06 dist-test-slave-s0b2 krb5kdc[27775](info): set up 2 sockets
Sep 25 01:28:06 dist-test-slave-s0b2 krb5kdc[27775](info): commencing operation
krb5kdc: starting...
W20260925 01:28:07.575099 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:07 dist-test-slave-s0b2 krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:07.813388 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:08.148200 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:08.148473 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:08.148573 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:08.155900 27788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:08.156014 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:08.156068 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:08.156128 27788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:08.156172 27788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:08.165926 27788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:08.167214 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:08.168514 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:08.179735 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:08.179757 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:08.179798 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:08.180941 27788 server_base.cc:1034] running on GCE node
I20260925 01:28:08.181739 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:08.183171 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:08.184379 27788 hybrid_clock.cc:648] HybridClock initialized: now 1790299688184326 us; error 46 us; skew 500 ppm
I20260925 01:28:08.184703 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:08 dist-test-slave-s0b2 krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:08.191604 27788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:08.192970 27788 webserver.cc:460] Webserver started at http://127.0.0.1:38987/ using document root <none> and password file <none>
I20260925 01:28:08.193620 27788 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:08.193717 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:08.193974 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:08.196859 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "625aa035b5ca4dcaa625c5ad2b0028c3"
format_stamp: "Formatted at 2026-09-25 01:28:08 on dist-test-slave-s0b2"
server_key: "671b54de1c8069637f9e47cd7bc8b80c"
server_key_iv: "40ab5aaeff7345356e4fee565f05c766"
server_key_version: "encryptionkey@0"
I20260925 01:28:08.197535 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "625aa035b5ca4dcaa625c5ad2b0028c3"
format_stamp: "Formatted at 2026-09-25 01:28:08 on dist-test-slave-s0b2"
server_key: "671b54de1c8069637f9e47cd7bc8b80c"
server_key_iv: "40ab5aaeff7345356e4fee565f05c766"
server_key_version: "encryptionkey@0"
I20260925 01:28:08.202407 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:28:08.205713 27803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:08.207062 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:08.207201 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "625aa035b5ca4dcaa625c5ad2b0028c3"
format_stamp: "Formatted at 2026-09-25 01:28:08 on dist-test-slave-s0b2"
server_key: "671b54de1c8069637f9e47cd7bc8b80c"
server_key_iv: "40ab5aaeff7345356e4fee565f05c766"
server_key_version: "encryptionkey@0"
I20260925 01:28:08.207355 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:08.217284 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:08.220897 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:08.221118 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:08.237661 27788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33761
I20260925 01:28:08.237679 27854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33761 every 8 connection(s)
I20260925 01:28:08.239012 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:08.242632 27855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:08.246464 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27788
I20260925 01:28:08.246632 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:08.247140 7090 external_mini_cluster.cc:1428] Setting key 4d317ef436aa434955b46de751e29226
I20260925 01:28:08.255220 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3: Bootstrap starting.
I20260925 01:28:08.258998 27855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:08.260326 27855 log.cc:824] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:28:08 dist-test-slave-s0b2 krb5kdc[27775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:28:08.263840 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3: No bootstrap required, opened a new log
I20260925 01:28:08.271354 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:08.248905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44565 (local address 127.0.0.1:33761)
0925 01:28:08.249634 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:08.249645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:08.249685 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:28:08.251311 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:08.251336 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:08.251771 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:08.252121 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:08.253822 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:08.255166 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:08.256916 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:08.257610 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:08.257772 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:08.262780 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:08.262791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:08.262813 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:08.262899 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:08.266816 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:08.267521 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:08.267530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:08.267539 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:08.267757 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:08.268093 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:08.268098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:08.268102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:08.268787 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:08.269070 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:08.269452 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 01:28:08.269924 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":333,"threads_started":1}
I20260925 01:28:08.273996 27855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260925 01:28:08.274461 27855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:08.274575 27855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625aa035b5ca4dcaa625c5ad2b0028c3, State: Initialized, Role: FOLLOWER
I20260925 01:28:08.275278 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:08.275230 27855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260925 01:28:08.275492 27855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:08.275619 27855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:08.275758 27855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:08.278980 27855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260925 01:28:08.279611 27855 leader_election.cc:302] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625aa035b5ca4dcaa625c5ad2b0028c3; no voters:
I20260925 01:28:08.280247 27855 leader_election.cc:288] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:08.280334 27864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:08.282500 27864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [term 1 LEADER]: Becoming Leader. State: Replica: 625aa035b5ca4dcaa625c5ad2b0028c3, State: Running, Role: LEADER
I20260925 01:28:08.283133 27864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } }
I20260925 01:28:08.283499 27855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:08.285957 27865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260925 01:28:08.286423 27865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:08.286950 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625aa035b5ca4dcaa625c5ad2b0028c3. Latest consensus state: current_term: 1 leader_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625aa035b5ca4dcaa625c5ad2b0028c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33761 } } }
I20260925 01:28:08.287220 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:08.287434 27871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:08.302618 27871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:08.306352 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:08.280911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42921 (local address 127.0.0.1:33761)
0925 01:28:08.281106 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:08.281114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:08.287638 (+ 6524us) server_negotiation.cc:407] Connection header received
0925 01:28:08.287868 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:08.287874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:08.287954 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:08.288049 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:08.290328 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:08.291584 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:08.293770 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:08.294613 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:08.294731 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:08.298474 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:08.298483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:08.298488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:08.298545 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:08.302542 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:08.303633 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:08.303639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:08.303642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:08.303809 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:08.304314 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:08.304319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:08.304322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:08.304772 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:08.305010 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:08.305679 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 01:28:08.305923 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:28:08.314224 27871 catalog_manager.cc:1347] Generated new cluster ID: c3c7a7ed2532464d927d4105fd435864
I20260925 01:28:08.314425 27871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:08.334830 27871 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:08.336205 27871 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:08.357398 27871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625aa035b5ca4dcaa625c5ad2b0028c3: Generated new TSK 0
I20260925 01:28:08.358271 27871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:08.367875 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27788
I20260925 01:28:08.368907 27850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:08.461470 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27788
2026-09-25T01:28:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:08 dist-test-slave-s0b2 krb5kdc[27887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:08 dist-test-slave-s0b2 krb5kdc[27887](info): set up 2 sockets
Sep 25 01:28:08 dist-test-slave-s0b2 krb5kdc[27887](info): commencing operation
krb5kdc: starting...
W20260925 01:28:09.471143 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:09 dist-test-slave-s0b2 krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:09.713572 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:10.030390 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:10.030679 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:10.030807 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:10.037837 27900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:10.037951 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:10.038033 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:10.038113 27900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:10.038174 27900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:10.047833 27900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:10.049059 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:10.050370 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:10.061369 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:10.061451 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:10.061419 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:10.062273 27900 server_base.cc:1034] running on GCE node
I20260925 01:28:10.062968 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:10.064280 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:10.065464 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790299690065415 us; error 44 us; skew 500 ppm
I20260925 01:28:10.065773 27900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:10 dist-test-slave-s0b2 krb5kdc[27887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:10.068722 27900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:10.070001 27900 webserver.cc:460] Webserver started at http://127.0.0.1:43323/ using document root <none> and password file <none>
I20260925 01:28:10.070667 27900 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:10.070766 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:10.071002 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:10.073796 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "95b6962c4cff410e9394c3d634111648"
format_stamp: "Formatted at 2026-09-25 01:28:10 on dist-test-slave-s0b2"
server_key: "f1f794c6197604128b82fc4084954bf2"
server_key_iv: "3995832fb848e5b8f5d75b16bb0ddbca"
server_key_version: "encryptionkey@0"
I20260925 01:28:10.074530 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "95b6962c4cff410e9394c3d634111648"
format_stamp: "Formatted at 2026-09-25 01:28:10 on dist-test-slave-s0b2"
server_key: "f1f794c6197604128b82fc4084954bf2"
server_key_iv: "3995832fb848e5b8f5d75b16bb0ddbca"
server_key_version: "encryptionkey@0"
I20260925 01:28:10.079423 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:28:10.082692 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:10.083976 27900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:28:10.084137 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "95b6962c4cff410e9394c3d634111648"
format_stamp: "Formatted at 2026-09-25 01:28:10 on dist-test-slave-s0b2"
server_key: "f1f794c6197604128b82fc4084954bf2"
server_key_iv: "3995832fb848e5b8f5d75b16bb0ddbca"
server_key_version: "encryptionkey@0"
I20260925 01:28:10.084329 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:10.108723 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:10.112788 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:10.113040 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:10.129683 27900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34589
I20260925 01:28:10.129712 27966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34589 every 8 connection(s)
I20260925 01:28:10.130999 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:10.134770 27967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:10.135181 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 27900
I20260925 01:28:10.135344 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:10.135762 7090 external_mini_cluster.cc:1428] Setting key dbddbeec335c2e38a1a8d66aaebf61d8
I20260925 01:28:10.146555 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648: Bootstrap starting.
I20260925 01:28:10.148594 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:10.137472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48007 (local address 127.0.0.1:34589)
0925 01:28:10.138200 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:10.138215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:10.138285 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:28:10.140563 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:10.140585 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:10.140975 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:10.141331 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:10.141895 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:10.143182 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:10.144495 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:10.145155 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:10.145320 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:10.146423 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:10.146430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:10.146445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:10.146977 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:10.147055 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:10.147079 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:10.147182 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:10.147404 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:28:10.147640 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":326,"threads_started":1}
I20260925 01:28:10.149667 27967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:10.150756 27967 log.cc:824] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:10.152050 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:10.153409 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648: No bootstrap required, opened a new log
I20260925 01:28:10.162082 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:10.156095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38429 (local address 127.0.0.1:34589)
0925 01:28:10.156265 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:10.156271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:10.156608 (+ 337us) server_negotiation.cc:407] Connection header received
0925 01:28:10.156748 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:10.156753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:10.156834 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:10.156945 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:10.157371 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:10.158457 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:10.159685 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:10.160380 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:10.160515 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:10.161428 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:10.161434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:10.161439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:10.161459 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:10.161504 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:10.161523 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:10.161630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:10.161698 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 01:28:10.161781 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:28:10.162348 27967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260925 01:28:10.162760 27967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:10.162882 27967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b6962c4cff410e9394c3d634111648, State: Initialized, Role: FOLLOWER
I20260925 01:28:10.163493 27967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260925 01:28:10.163712 27967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:10.163807 27967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:10.163926 27967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:10.165959 27967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260925 01:28:10.166462 27967 leader_election.cc:302] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95b6962c4cff410e9394c3d634111648; no voters:
I20260925 01:28:10.166993 27967 leader_election.cc:288] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:10.167099 27977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:10.167739 27977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [term 1 LEADER]: Becoming Leader. State: Replica: 95b6962c4cff410e9394c3d634111648, State: Running, Role: LEADER
I20260925 01:28:10.168206 27977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } }
I20260925 01:28:10.168956 27967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:10.170900 27978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95b6962c4cff410e9394c3d634111648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } } }
I20260925 01:28:10.170873 27979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95b6962c4cff410e9394c3d634111648. Latest consensus state: current_term: 1 leader_uuid: "95b6962c4cff410e9394c3d634111648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95b6962c4cff410e9394c3d634111648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34589 } } }
I20260925 01:28:10.171334 27978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:10.171334 27979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:10.172968 27983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:10.178450 27983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:10.188583 27983 catalog_manager.cc:1347] Generated new cluster ID: 6ef2f829366d49deaf685693b2355233
I20260925 01:28:10.188679 27983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:10.201835 27983 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:10.203230 27983 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:10.211637 27983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95b6962c4cff410e9394c3d634111648: Generated new TSK 0
I20260925 01:28:10.212397 27983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:10.217412 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27900
W20260925 01:28:10.218478 27962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48007
W20260925 01:28:10.219125 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 27900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:10.219273 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 27900
2026-09-25T01:28:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1756 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:10 dist-test-slave-s0b2 krb5kdc[27998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:10 dist-test-slave-s0b2 krb5kdc[27998](info): set up 2 sockets
Sep 25 01:28:10 dist-test-slave-s0b2 krb5kdc[27998](info): commencing operation
krb5kdc: starting...
W20260925 01:28:11.231142 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:11 dist-test-slave-s0b2 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:11.458951 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:11.772083 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:11.772356 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:11.772480 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:11.779500 28011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:11.779600 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:11.779673 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:11.779753 28011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:11.779808 28011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:11.789355 28011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:11.790669 28011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:11.791983 28011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:11.803449 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:11.803414 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:11.803375 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:11.804247 28011 server_base.cc:1034] running on GCE node
I20260925 01:28:11.805023 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:11.806401 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:11.807633 28011 hybrid_clock.cc:648] HybridClock initialized: now 1790299691807586 us; error 40 us; skew 500 ppm
I20260925 01:28:11.807969 28011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:11 dist-test-slave-s0b2 krb5kdc[27998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:11.810874 28011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:11.812389 28011 webserver.cc:460] Webserver started at http://127.0.0.1:44845/ using document root <none> and password file <none>
I20260925 01:28:11.813076 28011 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:11.813278 28011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:11.813573 28011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:11.816462 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "65798b96efac43c187a7061b69be6ffe"
format_stamp: "Formatted at 2026-09-25 01:28:11 on dist-test-slave-s0b2"
server_key: "1eea8278c5e9cb84de5773f7ff2cfa32"
server_key_iv: "201b52c65712a5c4326b5fc2b22eae30"
server_key_version: "encryptionkey@0"
I20260925 01:28:11.817206 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "65798b96efac43c187a7061b69be6ffe"
format_stamp: "Formatted at 2026-09-25 01:28:11 on dist-test-slave-s0b2"
server_key: "1eea8278c5e9cb84de5773f7ff2cfa32"
server_key_iv: "201b52c65712a5c4326b5fc2b22eae30"
server_key_version: "encryptionkey@0"
I20260925 01:28:11.822280 28011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:28:11.825590 28026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:11.826908 28011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:28:11.827083 28011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "65798b96efac43c187a7061b69be6ffe"
format_stamp: "Formatted at 2026-09-25 01:28:11 on dist-test-slave-s0b2"
server_key: "1eea8278c5e9cb84de5773f7ff2cfa32"
server_key_iv: "201b52c65712a5c4326b5fc2b22eae30"
server_key_version: "encryptionkey@0"
I20260925 01:28:11.827255 28011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:11.832471 28011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:11.836087 28011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:11.836352 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:11.854043 28011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44455
I20260925 01:28:11.854084 28077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44455 every 8 connection(s)
I20260925 01:28:11.855391 28011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:11.859067 28078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:11.859185 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28011
I20260925 01:28:11.859361 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:11.859931 7090 external_mini_cluster.cc:1428] Setting key 34c0a852efc3e1aef47d59ddd506d018
I20260925 01:28:11.867249 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:11.861690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50845 (local address 127.0.0.1:44455)
0925 01:28:11.862534 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:11.862546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:11.862589 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:28:11.864078 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:11.864100 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:11.864425 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:11.864806 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:11.865746 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:11.865753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:11.865765 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:11.865779 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:11.865842 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:11.865863 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:11.866074 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:11.866197 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 01:28:11.866394 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":334,"threads_started":1}
I20260925 01:28:11.870819 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:11.872051 28078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe: Bootstrap starting.
I20260925 01:28:11.875360 28078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:11.876466 28078 log.cc:824] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:11.877099 28081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:11.874844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42653 (local address 127.0.0.1:44455)
0925 01:28:11.874993 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:11.874999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:11.875341 (+ 342us) server_negotiation.cc:407] Connection header received
0925 01:28:11.875493 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:11.875496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:11.875542 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:11.875618 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:11.876470 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:11.876475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:11.876480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:11.876497 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:11.876533 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:11.876547 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:11.876638 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:11.876747 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 01:28:11.876803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:28:11.879190 28078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe: No bootstrap required, opened a new log
I20260925 01:28:11.886016 28078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260925 01:28:11.886394 28078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:11.886484 28078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65798b96efac43c187a7061b69be6ffe, State: Initialized, Role: FOLLOWER
I20260925 01:28:11.887099 28078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260925 01:28:11.887352 28078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:11.887488 28078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:11.887639 28078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:11.889660 28078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260925 01:28:11.890120 28078 leader_election.cc:302] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65798b96efac43c187a7061b69be6ffe; no voters:
I20260925 01:28:11.890678 28078 leader_election.cc:288] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:11.890815 28088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:11.891386 28088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [term 1 LEADER]: Becoming Leader. State: Replica: 65798b96efac43c187a7061b69be6ffe, State: Running, Role: LEADER
I20260925 01:28:11.891870 28088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } }
I20260925 01:28:11.892544 28078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:11.894269 28089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65798b96efac43c187a7061b69be6ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } } }
I20260925 01:28:11.894690 28089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:11.896441 28090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65798b96efac43c187a7061b69be6ffe. Latest consensus state: current_term: 1 leader_uuid: "65798b96efac43c187a7061b69be6ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65798b96efac43c187a7061b69be6ffe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44455 } } }
I20260925 01:28:11.896801 28090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:11.897477 28096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:11.902285 28096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:11.910045 28096 catalog_manager.cc:1347] Generated new cluster ID: ed35bea41c2b44ae99b7e45bfa5c4907
I20260925 01:28:11.910146 28096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:11.923161 28096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:11.923560 28096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:11.930358 28096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65798b96efac43c187a7061b69be6ffe: Generated new TSK 0
I20260925 01:28:11.931205 28096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:11.936389 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28011
W20260925 01:28:11.937754 28073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50845
W20260925 01:28:11.938441 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:11.938572 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28011
2026-09-25T01:28:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1719 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:12 dist-test-slave-s0b2 krb5kdc[28109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:12 dist-test-slave-s0b2 krb5kdc[28109](info): set up 2 sockets
Sep 25 01:28:12 dist-test-slave-s0b2 krb5kdc[28109](info): commencing operation
krb5kdc: starting...
W20260925 01:28:12.939047 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:13.165035 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:13.487176 28122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:13.487458 28122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:13.487597 28122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:13.494699 28122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:13.494820 28122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:13.494897 28122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:13.494976 28122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:13.495033 28122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:13.504587 28122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:13.505787 28122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:13.507050 28122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:13.518157 28127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:13.518397 28128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:13.518460 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:13.519192 28122 server_base.cc:1034] running on GCE node
I20260925 01:28:13.519855 28122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:13.521127 28122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:13.522337 28122 hybrid_clock.cc:648] HybridClock initialized: now 1790299693522301 us; error 30 us; skew 500 ppm
I20260925 01:28:13.522670 28122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:13.528998 28122 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:13.530431 28122 webserver.cc:460] Webserver started at http://127.0.0.1:40367/ using document root <none> and password file <none>
I20260925 01:28:13.531087 28122 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:13.531193 28122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:13.531461 28122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:13.534333 28122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f"
format_stamp: "Formatted at 2026-09-25 01:28:13 on dist-test-slave-s0b2"
server_key: "75c2289778eec772686ac9c08aa99365"
server_key_iv: "b6f027954ad8e5c5e147871d9da5b24f"
server_key_version: "encryptionkey@0"
I20260925 01:28:13.535071 28122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f"
format_stamp: "Formatted at 2026-09-25 01:28:13 on dist-test-slave-s0b2"
server_key: "75c2289778eec772686ac9c08aa99365"
server_key_iv: "b6f027954ad8e5c5e147871d9da5b24f"
server_key_version: "encryptionkey@0"
I20260925 01:28:13.539927 28122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:28:13.543268 28137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:13.544562 28122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:28:13.544770 28122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f"
format_stamp: "Formatted at 2026-09-25 01:28:13 on dist-test-slave-s0b2"
server_key: "75c2289778eec772686ac9c08aa99365"
server_key_iv: "b6f027954ad8e5c5e147871d9da5b24f"
server_key_version: "encryptionkey@0"
I20260925 01:28:13.544946 28122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:13.562196 28122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:13.566346 28122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:13.566610 28122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:13.583742 28122 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40365
I20260925 01:28:13.583765 28188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40365 every 8 connection(s)
I20260925 01:28:13.585058 28122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:13.588742 28189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:13.589843 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28122
I20260925 01:28:13.590001 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:13.590474 7090 external_mini_cluster.cc:1428] Setting key 5fe802bd52c4ed584240e3eaa083b94f
I20260925 01:28:13.602070 28189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f: Bootstrap starting.
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:28:13.605813 28189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:13.606979 28189 log.cc:824] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:13.609982 28189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f: No bootstrap required, opened a new log
I20260925 01:28:13.613521 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:13.592244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:59455 (local address 127.0.0.1:40365)
0925 01:28:13.592977 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:13.592989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:13.593029 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:28:13.594673 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:13.594695 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:13.595099 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:13.595417 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:13.597106 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:13.598461 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:13.600302 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:13.600981 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:13.601139 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:13.605974 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:13.605985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:13.606002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:13.606042 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:13.609268 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:13.610164 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:13.610171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:13.610177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:13.610378 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:13.610900 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:13.610904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:13.610906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:13.611420 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:13.611670 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:13.612194 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 01:28:13.612512 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":319,"threads_started":1}
I20260925 01:28:13.617229 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:13.618510 28189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260925 01:28:13.618908 28189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:13.619005 28189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c27ff1891b4e69bf0f6ba82d2dc30f, State: Initialized, Role: FOLLOWER
I20260925 01:28:13.619601 28189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260925 01:28:13.619879 28189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:13.620011 28189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:13.620177 28189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:13.622982 28189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260925 01:28:13.623476 28189 leader_election.cc:302] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c27ff1891b4e69bf0f6ba82d2dc30f; no voters:
I20260925 01:28:13.623999 28189 leader_election.cc:288] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:13.624146 28199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:13.625890 28189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:13.626524 28199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [term 1 LEADER]: Becoming Leader. State: Replica: 23c27ff1891b4e69bf0f6ba82d2dc30f, State: Running, Role: LEADER
I20260925 01:28:13.627112 28199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } }
I20260925 01:28:13.629555 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260925 01:28:13.629900 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:13.630388 28201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c27ff1891b4e69bf0f6ba82d2dc30f. Latest consensus state: current_term: 1 leader_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c27ff1891b4e69bf0f6ba82d2dc30f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40365 } } }
I20260925 01:28:13.630703 28201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:13.639120 28205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:13.639991 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:13.622191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:43493 (local address 127.0.0.1:40365)
0925 01:28:13.622386 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:13.622392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:13.622775 (+ 383us) server_negotiation.cc:407] Connection header received
0925 01:28:13.622901 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:13.622906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:13.622990 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:13.623098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:13.624644 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:13.625865 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:13.627743 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:13.628435 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:13.628567 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:13.632303 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:13.632311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:13.632317 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:13.632359 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:13.636080 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:13.637111 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:13.637117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:13.637120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:13.637276 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:13.637873 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:13.637878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:13.637881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:13.638354 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:13.638632 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:13.639405 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 01:28:13.639590 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:28:13.643956 28205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:13.652431 28205 catalog_manager.cc:1347] Generated new cluster ID: b72dcb64396346aaa4f02742fabee654
I20260925 01:28:13.652535 28205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:13.676638 28205 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:13.678593 28205 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:13.697232 28205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c27ff1891b4e69bf0f6ba82d2dc30f: Generated new TSK 0
I20260925 01:28:13.698122 28205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:13.717110 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28122
I20260925 01:28:13.718607 28184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:13.810405 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28122
W20260925 01:28:13.825570 28107 connection.cc:535] client connection to 127.0.0.1:40365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40365): Connection reset by peer (error 104)
2026-09-25T01:28:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28222](info): set up 2 sockets
Sep 25 01:28:13 dist-test-slave-s0b2 krb5kdc[28222](info): commencing operation
krb5kdc: starting...
W20260925 01:28:14.827338 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:14 dist-test-slave-s0b2 krb5kdc[28222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:15.068782 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:15.385489 28235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:15.385812 28235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:15.385954 28235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:15.393169 28235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:15.393297 28235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:15.393376 28235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:15.393460 28235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:15.393517 28235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:15.403347 28235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:15.404659 28235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:15.405956 28235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:15.417608 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:15.417666 28243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:15.417646 28241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:15.418174 28235 server_base.cc:1034] running on GCE node
I20260925 01:28:15.419133 28235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:15.420529 28235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:15.421756 28235 hybrid_clock.cc:648] HybridClock initialized: now 1790299695421703 us; error 40 us; skew 500 ppm
I20260925 01:28:15.422116 28235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:15 dist-test-slave-s0b2 krb5kdc[28222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:15.425176 28235 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:15.426602 28235 webserver.cc:460] Webserver started at http://127.0.0.1:37833/ using document root <none> and password file <none>
I20260925 01:28:15.427273 28235 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:15.427392 28235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:15.427680 28235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:15.430717 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "aebac95a74c44b089f4e697bd2ede5e9"
format_stamp: "Formatted at 2026-09-25 01:28:15 on dist-test-slave-s0b2"
server_key: "6ef6c31a18e7260d131d09d20db7d0a3"
server_key_iv: "5006b02f8eb984d3a0bfa57248dec392"
server_key_version: "encryptionkey@0"
I20260925 01:28:15.431452 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "aebac95a74c44b089f4e697bd2ede5e9"
format_stamp: "Formatted at 2026-09-25 01:28:15 on dist-test-slave-s0b2"
server_key: "6ef6c31a18e7260d131d09d20db7d0a3"
server_key_iv: "5006b02f8eb984d3a0bfa57248dec392"
server_key_version: "encryptionkey@0"
I20260925 01:28:15.436594 28235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:28:15.440119 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:15.441463 28235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:28:15.441682 28235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "aebac95a74c44b089f4e697bd2ede5e9"
format_stamp: "Formatted at 2026-09-25 01:28:15 on dist-test-slave-s0b2"
server_key: "6ef6c31a18e7260d131d09d20db7d0a3"
server_key_iv: "5006b02f8eb984d3a0bfa57248dec392"
server_key_version: "encryptionkey@0"
I20260925 01:28:15.441867 28235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:15.457146 28235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:15.461277 28235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:15.461596 28235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:15.478834 28235 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42091
I20260925 01:28:15.478864 28301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42091 every 8 connection(s)
I20260925 01:28:15.480180 28235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:15.480899 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28235
I20260925 01:28:15.481072 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:15.481633 7090 external_mini_cluster.cc:1428] Setting key 44dce93032cd0c27393723f8279dfa89
I20260925 01:28:15.484624 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:15.494402 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:15.483552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:55801 (local address 127.0.0.1:42091)
0925 01:28:15.484269 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:15.484281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:15.484319 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:28:15.485919 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:15.485940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:15.486373 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:15.486715 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:15.487266 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:15.488614 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:15.489973 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:15.490690 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:15.490852 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:15.492111 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:15.492119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:15.492133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:15.492715 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:15.492802 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:15.492830 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:15.492980 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:15.493260 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 01:28:15.493427 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":290,"threads_started":1}
I20260925 01:28:15.495671 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9: Bootstrap starting.
I20260925 01:28:15.498363 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:15.498978 28302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:15.499951 28302 log.cc:824] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:15.503400 28302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9: No bootstrap required, opened a new log
I20260925 01:28:15.509418 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:15.502808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:50425 (local address 127.0.0.1:42091)
0925 01:28:15.503000 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:15.503006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:15.503355 (+ 349us) server_negotiation.cc:407] Connection header received
0925 01:28:15.503584 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:15.503588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:15.503650 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:15.503743 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:15.504355 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:15.505129 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:15.506956 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:15.507412 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:15.507510 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:15.508685 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:15.508697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:15.508700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:15.508721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:15.508761 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:15.508775 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:15.508866 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:15.509043 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 01:28:15.509095 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:28:15.512475 28302 raft_consensus.cc:348] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260925 01:28:15.512859 28302 raft_consensus.cc:374] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:15.512957 28302 raft_consensus.cc:729] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebac95a74c44b089f4e697bd2ede5e9, State: Initialized, Role: FOLLOWER
I20260925 01:28:15.513643 28302 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260925 01:28:15.513890 28302 raft_consensus.cc:388] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:15.514034 28302 raft_consensus.cc:482] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:15.514227 28302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:15.516426 28302 raft_consensus.cc:504] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260925 01:28:15.516981 28302 leader_election.cc:302] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aebac95a74c44b089f4e697bd2ede5e9; no voters:
I20260925 01:28:15.517623 28302 leader_election.cc:288] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:15.517724 28312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:15.518391 28312 raft_consensus.cc:686] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [term 1 LEADER]: Becoming Leader. State: Replica: aebac95a74c44b089f4e697bd2ede5e9, State: Running, Role: LEADER
I20260925 01:28:15.518929 28312 consensus_queue.cc:237] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260925 01:28:15.519852 28302 sys_catalog.cc:564] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:15.521601 28313 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aebac95a74c44b089f4e697bd2ede5e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } } }
I20260925 01:28:15.522064 28313 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:15.522718 28314 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aebac95a74c44b089f4e697bd2ede5e9. Latest consensus state: current_term: 1 leader_uuid: "aebac95a74c44b089f4e697bd2ede5e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebac95a74c44b089f4e697bd2ede5e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } } }
I20260925 01:28:15.523020 28314 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:15.527473 28320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:15.532881 28320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:15.541272 28320 catalog_manager.cc:1347] Generated new cluster ID: 7fcfb70c25ce47afbadd0b2f35a78e83
I20260925 01:28:15.541385 28320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:15.555651 28320 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:15.557493 28320 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:15.568368 28320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aebac95a74c44b089f4e697bd2ede5e9: Generated new TSK 0
I20260925 01:28:15.569607 28320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:15.576659 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28235
W20260925 01:28:15.577973 28297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:55801
W20260925 01:28:15.578737 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:15.578871 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28235
W20260925 01:28:15.593207 28220 connection.cc:535] client connection to 127.0.0.1:42091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42091): Connection reset by peer (error 104)
2026-09-25T01:28:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1767 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:15 dist-test-slave-s0b2 krb5kdc[28333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:15 dist-test-slave-s0b2 krb5kdc[28333](info): set up 2 sockets
Sep 25 01:28:15 dist-test-slave-s0b2 krb5kdc[28333](info): commencing operation
krb5kdc: starting...
W20260925 01:28:16.583139 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:16 dist-test-slave-s0b2 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:16.821702 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:17.137444 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:17.137714 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:17.137810 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:17.144909 28346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:17.145013 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:17.145066 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:17.145123 28346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:17.145169 28346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:17.154838 28346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:17.156049 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:17.157291 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:17.168349 28351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:17.168251 28352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:17.168321 28354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:17.169061 28346 server_base.cc:1034] running on GCE node
I20260925 01:28:17.169780 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:17.171113 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:17.172320 28346 hybrid_clock.cc:648] HybridClock initialized: now 1790299697172277 us; error 47 us; skew 500 ppm
I20260925 01:28:17.172637 28346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:17 dist-test-slave-s0b2 krb5kdc[28333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:17.175516 28346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:17.176932 28346 webserver.cc:460] Webserver started at http://127.0.0.1:38865/ using document root <none> and password file <none>
I20260925 01:28:17.177596 28346 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:17.177712 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:17.177991 28346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:17.180898 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "e737d361899340cdb8fe154a2984b51a"
format_stamp: "Formatted at 2026-09-25 01:28:17 on dist-test-slave-s0b2"
server_key: "350456dc8514f52d702dd20a7e230151"
server_key_iv: "12fa3f11ee87c737a57e2c67b3463d73"
server_key_version: "encryptionkey@0"
I20260925 01:28:17.181684 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "e737d361899340cdb8fe154a2984b51a"
format_stamp: "Formatted at 2026-09-25 01:28:17 on dist-test-slave-s0b2"
server_key: "350456dc8514f52d702dd20a7e230151"
server_key_iv: "12fa3f11ee87c737a57e2c67b3463d73"
server_key_version: "encryptionkey@0"
I20260925 01:28:17.186775 28346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:28:17.191282 28361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:17.192981 28346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:28:17.193156 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "e737d361899340cdb8fe154a2984b51a"
format_stamp: "Formatted at 2026-09-25 01:28:17 on dist-test-slave-s0b2"
server_key: "350456dc8514f52d702dd20a7e230151"
server_key_iv: "12fa3f11ee87c737a57e2c67b3463d73"
server_key_version: "encryptionkey@0"
I20260925 01:28:17.193365 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:17.199268 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:17.203055 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:17.203313 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:17.219523 28346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33703
I20260925 01:28:17.219560 28412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33703 every 8 connection(s)
I20260925 01:28:17.220796 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:17.223846 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28346
I20260925 01:28:17.223984 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:17.224407 7090 external_mini_cluster.cc:1428] Setting key 1f2e7cf6af3edf075a07f82054092b7b
I20260925 01:28:17.224512 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:17.232913 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:17.226113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:58361 (local address 127.0.0.1:33703)
0925 01:28:17.226912 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:17.226926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:17.226973 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:28:17.229162 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:17.229190 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:17.229626 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:17.230043 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:17.231028 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:17.231037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:17.231058 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:17.231075 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:17.231138 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:17.231171 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:17.231403 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:17.231534 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 01:28:17.231729 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":284,"threads_started":1}
I20260925 01:28:17.236330 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:17.238615 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a: Bootstrap starting.
I20260925 01:28:17.242425 28413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:17.243542 28413 log.cc:824] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:17.243700 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:17.241442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:42533 (local address 127.0.0.1:33703)
0925 01:28:17.241636 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:17.241641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:17.241944 (+ 303us) server_negotiation.cc:407] Connection header received
0925 01:28:17.242070 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:17.242075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:17.242133 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:17.242286 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:17.243118 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:17.243124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:17.243128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:17.243145 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:17.243189 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:17.243214 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:17.243344 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:17.243389 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 01:28:17.243442 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:28:17.246398 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a: No bootstrap required, opened a new log
I20260925 01:28:17.253485 28413 raft_consensus.cc:348] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } }
I20260925 01:28:17.253818 28413 raft_consensus.cc:374] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:17.253887 28413 raft_consensus.cc:729] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e737d361899340cdb8fe154a2984b51a, State: Initialized, Role: FOLLOWER
I20260925 01:28:17.254441 28413 consensus_queue.cc:260] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } }
I20260925 01:28:17.254647 28413 raft_consensus.cc:388] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:17.254750 28413 raft_consensus.cc:482] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:17.254869 28413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:17.256855 28413 raft_consensus.cc:504] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } }
I20260925 01:28:17.257272 28413 leader_election.cc:302] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e737d361899340cdb8fe154a2984b51a; no voters:
I20260925 01:28:17.257869 28413 leader_election.cc:288] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:17.257980 28423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:17.258622 28423 raft_consensus.cc:686] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [term 1 LEADER]: Becoming Leader. State: Replica: e737d361899340cdb8fe154a2984b51a, State: Running, Role: LEADER
I20260925 01:28:17.259141 28423 consensus_queue.cc:237] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } }
I20260925 01:28:17.260033 28413 sys_catalog.cc:564] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:17.261863 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e737d361899340cdb8fe154a2984b51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } } }
I20260925 01:28:17.262290 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:17.262547 28425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e737d361899340cdb8fe154a2984b51a. Latest consensus state: current_term: 1 leader_uuid: "e737d361899340cdb8fe154a2984b51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e737d361899340cdb8fe154a2984b51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33703 } } }
I20260925 01:28:17.263115 28425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:17.264432 28431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:17.269608 28431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:17.278219 28431 catalog_manager.cc:1347] Generated new cluster ID: 0a5d364087b8418d8236a1e3cb76f5ed
I20260925 01:28:17.278349 28431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:17.299194 28431 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:17.299619 28431 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:17.314590 28431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e737d361899340cdb8fe154a2984b51a: Generated new TSK 0
I20260925 01:28:17.315398 28431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:17.326519 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28346
W20260925 01:28:17.327566 28408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:58361
W20260925 01:28:17.328271 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:17.328392 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28346
2026-09-25T01:28:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1749 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:17 dist-test-slave-s0b2 krb5kdc[28444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:17 dist-test-slave-s0b2 krb5kdc[28444](info): set up 2 sockets
Sep 25 01:28:17 dist-test-slave-s0b2 krb5kdc[28444](info): commencing operation
krb5kdc: starting...
W20260925 01:28:18.347702 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:18 dist-test-slave-s0b2 krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:18.585341 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:18.910274 28457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:18.910602 28457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:18.910753 28457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:18.918219 28457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:18.918375 28457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:18.918460 28457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:18.918545 28457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:18.918610 28457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:18.928383 28457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:18.929551 28457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:18.930804 28457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:18.941844 28465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:18.941880 28463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:18.941918 28462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:18.942531 28457 server_base.cc:1034] running on GCE node
I20260925 01:28:18.943272 28457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:18.944608 28457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:18.945813 28457 hybrid_clock.cc:648] HybridClock initialized: now 1790299698945753 us; error 46 us; skew 500 ppm
I20260925 01:28:18.946142 28457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:18 dist-test-slave-s0b2 krb5kdc[28444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:18.955423 28457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:18.956916 28457 webserver.cc:460] Webserver started at http://127.0.0.1:45133/ using document root <none> and password file <none>
I20260925 01:28:18.957607 28457 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:18.957706 28457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:18.957985 28457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:18.960944 28457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "df2965ec0a35470d95d1c50c134808f2"
format_stamp: "Formatted at 2026-09-25 01:28:18 on dist-test-slave-s0b2"
server_key: "fa6533eefed55e274a725d0af92554a2"
server_key_iv: "611d047c1a7c09fa83c1c58236ab6af3"
server_key_version: "encryptionkey@0"
I20260925 01:28:18.961689 28457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "df2965ec0a35470d95d1c50c134808f2"
format_stamp: "Formatted at 2026-09-25 01:28:18 on dist-test-slave-s0b2"
server_key: "fa6533eefed55e274a725d0af92554a2"
server_key_iv: "611d047c1a7c09fa83c1c58236ab6af3"
server_key_version: "encryptionkey@0"
I20260925 01:28:18.966666 28457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:28:18.969985 28472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:18.971294 28457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:18.971446 28457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "df2965ec0a35470d95d1c50c134808f2"
format_stamp: "Formatted at 2026-09-25 01:28:18 on dist-test-slave-s0b2"
server_key: "fa6533eefed55e274a725d0af92554a2"
server_key_iv: "611d047c1a7c09fa83c1c58236ab6af3"
server_key_version: "encryptionkey@0"
I20260925 01:28:18.971613 28457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:18.984176 28457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:18.987869 28457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:18.988111 28457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:19.005179 28457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38653
I20260925 01:28:19.005220 28523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38653 every 8 connection(s)
I20260925 01:28:19.006512 28457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:19.006951 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28457
I20260925 01:28:19.007109 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:19.007658 7090 external_mini_cluster.cc:1428] Setting key d04f19c4d4ff740d60587720d30f7e88
I20260925 01:28:19.010942 28524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:28:19 dist-test-slave-s0b2 krb5kdc[28444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:28:19.022902 28524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2: Bootstrap starting.
I20260925 01:28:19.025986 28524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:19.027228 28524 log.cc:824] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:19.030366 28524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2: No bootstrap required, opened a new log
I20260925 01:28:19.030696 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:19.009423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:44789 (local address 127.0.0.1:38653)
0925 01:28:19.010157 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:19.010171 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:19.010218 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:28:19.012411 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:19.012437 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:19.012912 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:19.013317 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:19.014607 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:19.016630 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:19.017871 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:19.018901 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:19.019075 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:19.023176 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:19.023184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:19.023197 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:19.023237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:19.026281 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:19.027225 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:19.027231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:19.027237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:19.027441 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:19.027936 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:19.027941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:19.027948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:19.028650 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:19.028877 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:19.029359 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 01:28:19.029611 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":300,"threads_started":1}
I20260925 01:28:19.034592 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:19.038403 28524 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 01:28:19.038806 28524 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:19.038911 28524 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2965ec0a35470d95d1c50c134808f2, State: Initialized, Role: FOLLOWER
I20260925 01:28:19.039449 28524 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 01:28:19.039664 28524 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:19.039789 28524 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:19.039937 28524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:19.041978 28524 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 01:28:19.042514 28524 leader_election.cc:302] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2965ec0a35470d95d1c50c134808f2; no voters:
I20260925 01:28:19.043418 28524 leader_election.cc:288] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:19.043488 28534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:19.045113 28534 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [term 1 LEADER]: Becoming Leader. State: Replica: df2965ec0a35470d95d1c50c134808f2, State: Running, Role: LEADER
I20260925 01:28:19.045620 28524 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:19.045722 28534 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } }
I20260925 01:28:19.054399 28536 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2965ec0a35470d95d1c50c134808f2. Latest consensus state: current_term: 1 leader_uuid: "df2965ec0a35470d95d1c50c134808f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } } }
I20260925 01:28:19.054792 28536 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:19.055156 28535 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2965ec0a35470d95d1c50c134808f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2965ec0a35470d95d1c50c134808f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38653 } } }
I20260925 01:28:19.055413 28535 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:19.059294 28527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:19.038847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:41669 (local address 127.0.0.1:38653)
0925 01:28:19.039076 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:19.039082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:19.042767 (+ 3685us) server_negotiation.cc:407] Connection header received
0925 01:28:19.042946 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:19.042950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:19.043013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:19.043092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:19.044902 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:19.046136 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:19.047968 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:19.048682 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:19.048816 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:19.052021 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:19.052032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:19.052038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:19.052082 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:19.055939 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:19.056891 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:19.056898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:19.056902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:19.057085 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:19.057551 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:19.057556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:19.057559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:19.057961 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:19.058266 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:19.058675 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 01:28:19.058877 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 01:28:19.062879 28544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:19.067133 28544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:19.075054 28544 catalog_manager.cc:1347] Generated new cluster ID: 84b126c61c554d83a908179fcb0b94bd
I20260925 01:28:19.075165 28544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:19.097417 28544 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:19.099291 28544 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:19.107815 28544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2965ec0a35470d95d1c50c134808f2: Generated new TSK 0
I20260925 01:28:19.108826 28544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:19.115643 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28457
I20260925 01:28:19.116999 28519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:19.207923 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28457
W20260925 01:28:19.222343 28442 connection.cc:535] client connection to 127.0.0.1:38653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38653): Connection reset by peer (error 104)
2026-09-25T01:28:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:19 dist-test-slave-s0b2 krb5kdc[28556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:19 dist-test-slave-s0b2 krb5kdc[28556](info): set up 2 sockets
Sep 25 01:28:19 dist-test-slave-s0b2 krb5kdc[28556](info): commencing operation
krb5kdc: starting...
W20260925 01:28:20.223193 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:20 dist-test-slave-s0b2 krb5kdc[28556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:20.465807 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:20.785128 28569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:20.785391 28569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:20.785480 28569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:20.792567 28569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:20.792670 28569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:20.792728 28569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:20.792788 28569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:20.792829 28569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:20.802409 28569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:20.803601 28569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:20.804840 28569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:20.815781 28574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:20.815852 28577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:20.815877 28575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:20.817129 28569 server_base.cc:1034] running on GCE node
I20260925 01:28:20.817934 28569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:20.819547 28569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:20.820761 28569 hybrid_clock.cc:648] HybridClock initialized: now 1790299700820689 us; error 57 us; skew 500 ppm
I20260925 01:28:20.821108 28569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:20 dist-test-slave-s0b2 krb5kdc[28556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:20.824179 28569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:20.825627 28569 webserver.cc:460] Webserver started at http://127.0.0.1:44649/ using document root <none> and password file <none>
I20260925 01:28:20.826316 28569 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:20.826423 28569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:20.826681 28569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:20.829658 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "451c17f7ba7f45ec80c6f2d36c5a9161"
format_stamp: "Formatted at 2026-09-25 01:28:20 on dist-test-slave-s0b2"
server_key: "9b3f415888baa1b0df33d299305e9327"
server_key_iv: "74fb1d21a560c4dd5ea2c4bfecfb1b03"
server_key_version: "encryptionkey@0"
I20260925 01:28:20.830412 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "451c17f7ba7f45ec80c6f2d36c5a9161"
format_stamp: "Formatted at 2026-09-25 01:28:20 on dist-test-slave-s0b2"
server_key: "9b3f415888baa1b0df33d299305e9327"
server_key_iv: "74fb1d21a560c4dd5ea2c4bfecfb1b03"
server_key_version: "encryptionkey@0"
I20260925 01:28:20.835534 28569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:28:20.838840 28584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:20.840098 28569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:28:20.840281 28569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "451c17f7ba7f45ec80c6f2d36c5a9161"
format_stamp: "Formatted at 2026-09-25 01:28:20 on dist-test-slave-s0b2"
server_key: "9b3f415888baa1b0df33d299305e9327"
server_key_iv: "74fb1d21a560c4dd5ea2c4bfecfb1b03"
server_key_version: "encryptionkey@0"
I20260925 01:28:20.840454 28569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:20.852993 28569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:20.857076 28569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:20.857345 28569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:20.873871 28569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41431
I20260925 01:28:20.873914 28635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41431 every 8 connection(s)
I20260925 01:28:20.875519 28569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:20.879236 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:20.881974 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28569
I20260925 01:28:20.882117 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:20.882576 7090 external_mini_cluster.cc:1428] Setting key b1156b72a2908b9af519f8b31a74b90d
I20260925 01:28:20.890748 28636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161: Bootstrap starting.
I20260925 01:28:20.894217 28636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:20.895337 28636 log.cc:824] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:20.895579 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:20.884431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:33925 (local address 127.0.0.1:41431)
0925 01:28:20.885172 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:20.885183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:20.885224 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:28:20.886907 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:20.886930 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:20.887327 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:20.887639 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:20.888189 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:20.889473 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:20.891241 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:20.891902 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:20.892036 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:20.893284 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:20.893291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:20.893306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:20.893822 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:20.893902 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:20.893926 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:20.894039 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:20.894371 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 01:28:20.894567 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":325,"threads_started":1}
I20260925 01:28:20.898125 28636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161: No bootstrap required, opened a new log
I20260925 01:28:20.899356 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:20.907728 28636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260925 01:28:20.908064 28636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:20.908167 28636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c17f7ba7f45ec80c6f2d36c5a9161, State: Initialized, Role: FOLLOWER
I20260925 01:28:20.908840 28636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260925 01:28:20.909124 28636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:20.909255 28636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:20.909403 28636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:20.910662 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:20.904410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:43057 (local address 127.0.0.1:41431)
0925 01:28:20.904586 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:20.904591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:20.904883 (+ 292us) server_negotiation.cc:407] Connection header received
0925 01:28:20.905062 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:20.905068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:20.905144 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:20.905242 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:20.905645 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:20.906818 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:20.908127 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:20.908739 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:20.908866 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:20.909900 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:20.909907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:20.909912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:20.909938 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:20.910005 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:20.910028 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:20.910161 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:20.910289 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 01:28:20.910341 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:28:20.912425 28636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260925 01:28:20.912878 28636 leader_election.cc:302] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451c17f7ba7f45ec80c6f2d36c5a9161; no voters:
I20260925 01:28:20.913368 28636 leader_election.cc:288] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:20.913489 28646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:20.914156 28646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [term 1 LEADER]: Becoming Leader. State: Replica: 451c17f7ba7f45ec80c6f2d36c5a9161, State: Running, Role: LEADER
I20260925 01:28:20.914697 28646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260925 01:28:20.915263 28636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:20.917361 28647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260925 01:28:20.917384 28648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451c17f7ba7f45ec80c6f2d36c5a9161. Latest consensus state: current_term: 1 leader_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c17f7ba7f45ec80c6f2d36c5a9161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260925 01:28:20.917784 28647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:20.917837 28648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:20.918591 28653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:20.923664 28653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:20.932792 28653 catalog_manager.cc:1347] Generated new cluster ID: c521abeeeb854d4a85b4f94c93965bcb
I20260925 01:28:20.932921 28653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:20.945082 28653 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:20.946410 28653 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:20.953127 28653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451c17f7ba7f45ec80c6f2d36c5a9161: Generated new TSK 0
I20260925 01:28:20.954129 28653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:20.960289 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28569
W20260925 01:28:20.961620 28631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:33925
W20260925 01:28:20.962392 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:20.962550 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28569
W20260925 01:28:20.976713 28554 connection.cc:535] client connection to 127.0.0.1:41431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41431): Connection reset by peer (error 104)
2026-09-25T01:28:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1754 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:21 dist-test-slave-s0b2 krb5kdc[28667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:21 dist-test-slave-s0b2 krb5kdc[28667](info): set up 2 sockets
Sep 25 01:28:21 dist-test-slave-s0b2 krb5kdc[28667](info): commencing operation
krb5kdc: starting...
W20260925 01:28:21.971258 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:22 dist-test-slave-s0b2 krb5kdc[28667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:28:22.199537 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:22.515596 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:22.515863 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:22.515990 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:22.523175 28680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:22.523288 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:22.523363 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:22.523437 28680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:22.523494 28680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:22.533139 28680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:22.534376 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:22.535617 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:22.546451 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:22.546463 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:22.546595 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:22.547513 28680 server_base.cc:1034] running on GCE node
I20260925 01:28:22.548228 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:22.549564 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:22.550778 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790299702550722 us; error 46 us; skew 500 ppm
I20260925 01:28:22.551097 28680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:22 dist-test-slave-s0b2 krb5kdc[28667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:22.553972 28680 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:28:22.555358 28680 webserver.cc:460] Webserver started at http://127.0.0.1:34977/ using document root <none> and password file <none>
I20260925 01:28:22.555994 28680 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:22.556106 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:22.556383 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:22.559417 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "83068a3b5d154155a52ad60c5a05538b"
format_stamp: "Formatted at 2026-09-25 01:28:22 on dist-test-slave-s0b2"
server_key: "c36410c1731c8b6ed440e7a99b4ff5bc"
server_key_iv: "f9be5e8b043eaee5fe34a5bd6f5eeaf4"
server_key_version: "encryptionkey@0"
I20260925 01:28:22.560252 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "83068a3b5d154155a52ad60c5a05538b"
format_stamp: "Formatted at 2026-09-25 01:28:22 on dist-test-slave-s0b2"
server_key: "c36410c1731c8b6ed440e7a99b4ff5bc"
server_key_iv: "f9be5e8b043eaee5fe34a5bd6f5eeaf4"
server_key_version: "encryptionkey@0"
I20260925 01:28:22.565176 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 01:28:22.568543 28695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:22.569870 28680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:28:22.570027 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "83068a3b5d154155a52ad60c5a05538b"
format_stamp: "Formatted at 2026-09-25 01:28:22 on dist-test-slave-s0b2"
server_key: "c36410c1731c8b6ed440e7a99b4ff5bc"
server_key_iv: "f9be5e8b043eaee5fe34a5bd6f5eeaf4"
server_key_version: "encryptionkey@0"
I20260925 01:28:22.570206 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:22.575570 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:22.579274 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:22.579530 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:22.595893 28680 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44787
I20260925 01:28:22.595938 28746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44787 every 8 connection(s)
I20260925 01:28:22.597222 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:22.600785 28747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:22.601794 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28680
I20260925 01:28:22.601951 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:22.602432 7090 external_mini_cluster.cc:1428] Setting key e94e3aeb5936a144fe6acd83b165df96
I20260925 01:28:22.609817 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:22.604055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:36393 (local address 127.0.0.1:44787)
0925 01:28:22.604790 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:22.604801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:22.604840 (+ 39us) server_negotiation.cc:407] Connection header received
0925 01:28:22.606482 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:22.606504 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:22.606852 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:22.607173 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:22.608192 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:22.608198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:22.608211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:22.608225 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:22.608283 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:22.608301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:22.608459 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:22.608628 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 01:28:22.608812 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":306,"threads_started":1}
I20260925 01:28:22.613502 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:22.613863 28747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b: Bootstrap starting.
I20260925 01:28:22.616952 28747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:22.618100 28747 log.cc:824] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:22.619767 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:22.617449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:50967 (local address 127.0.0.1:44787)
0925 01:28:22.617618 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:22.617625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:22.618061 (+ 436us) server_negotiation.cc:407] Connection header received
0925 01:28:22.618194 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:22.618200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:22.618283 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:22.618384 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:22.619166 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:22.619171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:22.619176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:22.619202 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:22.619239 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:22.619253 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:22.619346 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:22.619458 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 01:28:22.619516 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:28:22.621058 28747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b: No bootstrap required, opened a new log
I20260925 01:28:22.628417 28747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260925 01:28:22.628785 28747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:22.628880 28747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83068a3b5d154155a52ad60c5a05538b, State: Initialized, Role: FOLLOWER
I20260925 01:28:22.629400 28747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260925 01:28:22.629628 28747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:22.629755 28747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:22.629901 28747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:22.632508 28747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260925 01:28:22.632982 28747 leader_election.cc:302] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83068a3b5d154155a52ad60c5a05538b; no voters:
I20260925 01:28:22.633512 28747 leader_election.cc:288] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:22.633649 28757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:22.634267 28757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [term 1 LEADER]: Becoming Leader. State: Replica: 83068a3b5d154155a52ad60c5a05538b, State: Running, Role: LEADER
I20260925 01:28:22.634872 28757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } }
I20260925 01:28:22.636018 28747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:22.637887 28759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83068a3b5d154155a52ad60c5a05538b. Latest consensus state: current_term: 1 leader_uuid: "83068a3b5d154155a52ad60c5a05538b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } } }
I20260925 01:28:22.638337 28759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:22.641139 28758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83068a3b5d154155a52ad60c5a05538b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83068a3b5d154155a52ad60c5a05538b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44787 } } }
I20260925 01:28:22.641384 28758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:22.642041 28765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:22.646199 28765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:22.654148 28765 catalog_manager.cc:1347] Generated new cluster ID: dc2b4c9ae6a04569bc661384f2b0a822
I20260925 01:28:22.654384 28765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:22.670322 28765 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:22.670710 28765 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:22.678970 28765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83068a3b5d154155a52ad60c5a05538b: Generated new TSK 0
I20260925 01:28:22.679952 28765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:22.692647 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28680
W20260925 01:28:22.693827 28742 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:36393
W20260925 01:28:22.694582 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:22.694747 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28680
2026-09-25T01:28:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:22 dist-test-slave-s0b2 krb5kdc[28778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:22 dist-test-slave-s0b2 krb5kdc[28778](info): set up 2 sockets
Sep 25 01:28:22 dist-test-slave-s0b2 krb5kdc[28778](info): commencing operation
krb5kdc: starting...
W20260925 01:28:23.719249 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:23 dist-test-slave-s0b2 krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:23.959141 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42383
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:24.274475 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:24.274756 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:24.274886 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:24.282002 28791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:24.282109 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:24.282183 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:24.282281 28791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:24.282341 28791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:24.291991 28791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:42383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:24.293279 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:24.294569 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:24.305642 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:24.305716 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:24.305711 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:24.306332 28791 server_base.cc:1034] running on GCE node
I20260925 01:28:24.307051 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:24.308382 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:24.309571 28791 hybrid_clock.cc:648] HybridClock initialized: now 1790299704309526 us; error 39 us; skew 500 ppm
I20260925 01:28:24.309909 28791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:24 dist-test-slave-s0b2 krb5kdc[28778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:24.317797 28791 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:24.319363 28791 webserver.cc:460] Webserver started at http://127.6.236.190:44391/ using document root <none> and password file <none>
I20260925 01:28:24.320051 28791 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:24.320160 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:24.320447 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:24.323412 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "6f7067f89f514c6686102c97d7564f16"
format_stamp: "Formatted at 2026-09-25 01:28:24 on dist-test-slave-s0b2"
server_key: "17fc708309908c879d73372900c05fc8"
server_key_iv: "665292fc511ede11bef4c9141a3368c6"
server_key_version: "encryptionkey@0"
I20260925 01:28:24.324151 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "6f7067f89f514c6686102c97d7564f16"
format_stamp: "Formatted at 2026-09-25 01:28:24 on dist-test-slave-s0b2"
server_key: "17fc708309908c879d73372900c05fc8"
server_key_iv: "665292fc511ede11bef4c9141a3368c6"
server_key_version: "encryptionkey@0"
I20260925 01:28:24.329110 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:28:24.332396 28806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:24.333618 28791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:28:24.333786 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "6f7067f89f514c6686102c97d7564f16"
format_stamp: "Formatted at 2026-09-25 01:28:24 on dist-test-slave-s0b2"
server_key: "17fc708309908c879d73372900c05fc8"
server_key_iv: "665292fc511ede11bef4c9141a3368c6"
server_key_version: "encryptionkey@0"
I20260925 01:28:24.333966 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:24.356166 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:24.360241 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:24.360510 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:24.377060 28791 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:42383
I20260925 01:28:24.377079 28857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:42383 every 8 connection(s)
I20260925 01:28:24.378419 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:24.380295 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28791
I20260925 01:28:24.380479 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:24.381021 7090 external_mini_cluster.cc:1428] Setting key 3dd65aa923baa6adb7591d032aea75e2
I20260925 01:28:24.382899 28858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:28:24 dist-test-slave-s0b2 krb5kdc[28778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:28:24.396625 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16: Bootstrap starting.
I20260925 01:28:24.400082 28858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:24.401387 28858 log.cc:824] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:24.404685 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16: No bootstrap required, opened a new log
I20260925 01:28:24.404980 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:24.382753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51801 (local address 127.6.236.190:42383)
0925 01:28:24.383522 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:24.383534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:24.383575 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:28:24.385538 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:24.385560 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:24.385955 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:24.386336 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:24.388121 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:24.389503 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:24.391262 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:24.391963 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:24.392112 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:24.397182 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:24.397191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:24.397209 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:24.397266 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:24.400336 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:24.401316 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:24.401323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:24.401330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:24.401538 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:24.402080 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:24.402085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:24.402088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:24.402870 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:24.403107 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:24.403602 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 01:28:24.403890 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":363,"threads_started":1}
I20260925 01:28:24.408730 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:24.413810 28858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } }
I20260925 01:28:24.414160 28858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:24.414310 28858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7067f89f514c6686102c97d7564f16, State: Initialized, Role: FOLLOWER
I20260925 01:28:24.414870 28858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } }
I20260925 01:28:24.415081 28858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:24.415205 28858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:24.415355 28858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:24.417505 28858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } }
I20260925 01:28:24.417956 28858 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7067f89f514c6686102c97d7564f16; no voters:
I20260925 01:28:24.418573 28858 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:24.418669 28868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:24.420763 28858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:24.421315 28868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7067f89f514c6686102c97d7564f16, State: Running, Role: LEADER
I20260925 01:28:24.421953 28868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } }
I20260925 01:28:24.425554 28870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7067f89f514c6686102c97d7564f16. Latest consensus state: current_term: 1 leader_uuid: "6f7067f89f514c6686102c97d7564f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } } }
I20260925 01:28:24.426095 28869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7067f89f514c6686102c97d7564f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7067f89f514c6686102c97d7564f16" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42383 } } }
I20260925 01:28:24.426477 28870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:24.426870 28869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:24.428982 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:24.413678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35421 (local address 127.6.236.190:42383)
0925 01:28:24.413855 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:24.413861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:24.414234 (+ 373us) server_negotiation.cc:407] Connection header received
0925 01:28:24.414413 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:24.414418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:24.414476 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:24.414553 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:24.416041 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:24.416837 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:24.418040 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:24.418584 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:24.418702 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:24.421174 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:24.421180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:24.421185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:24.421235 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:24.424864 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:24.425899 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:24.425904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:24.425908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:24.426089 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:24.426745 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:24.426753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:24.426756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:24.427185 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:24.428208 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:24.428391 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 01:28:24.428576 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:28:24.430510 28879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:24.434882 28879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:24.443575 28879 catalog_manager.cc:1347] Generated new cluster ID: 436e14f528db498f8bb01a365efcd7f4
I20260925 01:28:24.443668 28879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:24.454465 28879 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:24.455677 28879 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:24.463935 28879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7067f89f514c6686102c97d7564f16: Generated new TSK 0
I20260925 01:28:24.464671 28879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:24.471832 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28791
I20260925 01:28:24.473431 28853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:24.561132 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28791
W20260925 01:28:24.576272 28776 connection.cc:535] client connection to 127.6.236.190:42383 recv error: Network error: failed to read from TLS socket (remote: 127.6.236.190:42383): Connection reset by peer (error 104)
2026-09-25T01:28:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:24 dist-test-slave-s0b2 krb5kdc[28890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:24 dist-test-slave-s0b2 krb5kdc[28890](info): set up 2 sockets
Sep 25 01:28:24 dist-test-slave-s0b2 krb5kdc[28890](info): commencing operation
krb5kdc: starting...
W20260925 01:28:25.591303 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:25 dist-test-slave-s0b2 krb5kdc[28890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:25.831632 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40397
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:26.147423 28903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:26.147729 28903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:26.147863 28903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:26.154923 28903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:26.155033 28903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:26.155108 28903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:26.155181 28903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:26.155236 28903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:26.164908 28903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:40397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:26.166222 28903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:26.167531 28903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:26.182546 28908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:26.182580 28911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:26.182602 28909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:26.183446 28903 server_base.cc:1034] running on GCE node
I20260925 01:28:26.184206 28903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:26.185585 28903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:26.186779 28903 hybrid_clock.cc:648] HybridClock initialized: now 1790299706186742 us; error 28 us; skew 500 ppm
I20260925 01:28:26.187100 28903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:26 dist-test-slave-s0b2 krb5kdc[28890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299706, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:26.190016 28903 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:26.191453 28903 webserver.cc:460] Webserver started at http://127.6.236.190:45665/ using document root <none> and password file <none>
I20260925 01:28:26.192108 28903 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:26.192225 28903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:26.192512 28903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:26.195520 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "20225e0c1152405a9db96cc834dcc4c0"
format_stamp: "Formatted at 2026-09-25 01:28:26 on dist-test-slave-s0b2"
server_key: "ddb7173eb8454eebe3f1f23a2177b932"
server_key_iv: "807d7530b31c496075ce0ef32611c43a"
server_key_version: "encryptionkey@0"
I20260925 01:28:26.196267 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "20225e0c1152405a9db96cc834dcc4c0"
format_stamp: "Formatted at 2026-09-25 01:28:26 on dist-test-slave-s0b2"
server_key: "ddb7173eb8454eebe3f1f23a2177b932"
server_key_iv: "807d7530b31c496075ce0ef32611c43a"
server_key_version: "encryptionkey@0"
I20260925 01:28:26.201401 28903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 01:28:26.204845 28918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:26.206300 28903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:28:26.206482 28903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "20225e0c1152405a9db96cc834dcc4c0"
format_stamp: "Formatted at 2026-09-25 01:28:26 on dist-test-slave-s0b2"
server_key: "ddb7173eb8454eebe3f1f23a2177b932"
server_key_iv: "807d7530b31c496075ce0ef32611c43a"
server_key_version: "encryptionkey@0"
I20260925 01:28:26.206681 28903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:26.232327 28903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:26.236632 28903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:26.236912 28903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:26.253995 28903 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:40397
I20260925 01:28:26.254015 28969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:40397 every 8 connection(s)
I20260925 01:28:26.255342 28903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:26.259042 28970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:26.264020 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 28903
I20260925 01:28:26.264197 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:26.264703 7090 external_mini_cluster.cc:1428] Setting key f79d3d14926f64c1c9dbd8100b5d9318
I20260925 01:28:26.269853 28970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: Bootstrap starting.
I20260925 01:28:26.273448 28970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:26.274595 28970 log.cc:824] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:26.277515 28970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: No bootstrap required, opened a new log
I20260925 01:28:26.277822 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:26.266540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34825 (local address 127.6.236.190:40397)
0925 01:28:26.267349 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:26.267364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:26.267406 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:28:26.269067 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:26.269090 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:26.269477 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:26.269793 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:26.270361 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:26.271654 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:26.273417 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:26.274121 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:26.274290 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:26.275508 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:26.275518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:26.275537 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:26.276175 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:26.276250 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:26.276280 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:26.276395 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:26.276662 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 01:28:26.276864 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":362,"threads_started":1}
I20260925 01:28:26.281457 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:26.286476 28970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } }
I20260925 01:28:26.286835 28970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:26.286929 28970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20225e0c1152405a9db96cc834dcc4c0, State: Initialized, Role: FOLLOWER
I20260925 01:28:26.287545 28970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } }
I20260925 01:28:26.287782 28970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:26.287935 28970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:26.288108 28970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:26.290684 28970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } }
I20260925 01:28:26.291229 28970 leader_election.cc:302] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20225e0c1152405a9db96cc834dcc4c0; no voters:
I20260925 01:28:26.291842 28970 leader_election.cc:288] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:26.291983 28980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:26.293381 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:26.286300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52667 (local address 127.6.236.190:40397)
0925 01:28:26.286463 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:26.286470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:26.286910 (+ 440us) server_negotiation.cc:407] Connection header received
0925 01:28:26.287110 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:26.287115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:26.287188 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:26.287289 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:26.287752 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:26.288817 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:26.290066 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:26.290739 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:26.290865 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:26.292151 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:26.292160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:26.292166 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:26.292191 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:26.292277 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:26.292298 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:26.292959 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:26.293029 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 01:28:26.293080 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:28:26.293956 28970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:26.298471 28980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 20225e0c1152405a9db96cc834dcc4c0, State: Running, Role: LEADER
I20260925 01:28:26.298946 28980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } }
I20260925 01:28:26.301812 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20225e0c1152405a9db96cc834dcc4c0. Latest consensus state: current_term: 1 leader_uuid: "20225e0c1152405a9db96cc834dcc4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } } }
I20260925 01:28:26.302117 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:26.302652 28981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20225e0c1152405a9db96cc834dcc4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20225e0c1152405a9db96cc834dcc4c0" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 40397 } } }
I20260925 01:28:26.303107 28981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0 [sys.catalog]: This master's current role is: LEADER
W20260925 01:28:26.305852 28994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:28:26.305955 28994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:28:26.306075 28995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:26.307646 28995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:26.316228 28995 catalog_manager.cc:1347] Generated new cluster ID: f25e4ef7900447308b73941515ae5496
I20260925 01:28:26.316344 28995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:26.340063 28995 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:26.341367 28995 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:26.358040 28995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20225e0c1152405a9db96cc834dcc4c0: Generated new TSK 0
I20260925 01:28:26.358908 28995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:26.367681 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28903
W20260925 01:28:26.368871 28965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34825
W20260925 01:28:26.369552 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 28903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:26.369711 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 28903
W20260925 01:28:26.384222 28888 connection.cc:535] client connection to 127.6.236.190:40397 recv error: Network error: failed to read from TLS socket (remote: 127.6.236.190:40397): Connection reset by peer (error 104)
2026-09-25T01:28:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:26 dist-test-slave-s0b2 krb5kdc[29001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:26 dist-test-slave-s0b2 krb5kdc[29001](info): set up 2 sockets
Sep 25 01:28:26 dist-test-slave-s0b2 krb5kdc[29001](info): commencing operation
krb5kdc: starting...
W20260925 01:28:27.383366 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:27 dist-test-slave-s0b2 krb5kdc[29001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:27.625094 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33875
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:33875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:27.945551 29014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:27.945840 29014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:27.945976 29014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:27.953034 29014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:27.953148 29014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:27.953226 29014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:27.953305 29014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:27.953369 29014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:27.962939 29014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:33875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:27.964155 29014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:27.965376 29014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:27.976038 29019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:27.976125 29022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:27.976109 29020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:27.977720 29014 server_base.cc:1034] running on GCE node
I20260925 01:28:27.978430 29014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:27.979714 29014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:27.980911 29014 hybrid_clock.cc:648] HybridClock initialized: now 1790299707980867 us; error 36 us; skew 500 ppm
I20260925 01:28:27.981213 29014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:27 dist-test-slave-s0b2 krb5kdc[29001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:27.984174 29014 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:27.985492 29014 webserver.cc:460] Webserver started at http://127.6.236.190:37345/ using document root <none> and password file <none>
I20260925 01:28:27.986128 29014 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:27.986264 29014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:27.986550 29014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:27.989349 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "6fd7fe49eb49458d94b908dd099a76e4"
format_stamp: "Formatted at 2026-09-25 01:28:27 on dist-test-slave-s0b2"
server_key: "8d7c56d57cfd6b92e4e65c196a0722b5"
server_key_iv: "2376513ce4e9de872fbc9fecbf433ac4"
server_key_version: "encryptionkey@0"
I20260925 01:28:27.990083 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "6fd7fe49eb49458d94b908dd099a76e4"
format_stamp: "Formatted at 2026-09-25 01:28:27 on dist-test-slave-s0b2"
server_key: "8d7c56d57cfd6b92e4e65c196a0722b5"
server_key_iv: "2376513ce4e9de872fbc9fecbf433ac4"
server_key_version: "encryptionkey@0"
I20260925 01:28:27.994980 29014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 01:28:27.998163 29029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:27.999456 29014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:28:27.999632 29014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "6fd7fe49eb49458d94b908dd099a76e4"
format_stamp: "Formatted at 2026-09-25 01:28:27 on dist-test-slave-s0b2"
server_key: "8d7c56d57cfd6b92e4e65c196a0722b5"
server_key_iv: "2376513ce4e9de872fbc9fecbf433ac4"
server_key_version: "encryptionkey@0"
I20260925 01:28:27.999816 29014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:28.005352 29014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:28.008991 29014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:28.009240 29014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:28.025494 29014 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:33875
I20260925 01:28:28.025525 29080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:33875 every 8 connection(s)
I20260925 01:28:28.026822 29014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:28.027810 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29014
I20260925 01:28:28.027976 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:28.028445 7090 external_mini_cluster.cc:1428] Setting key a7567cff56d741b8cecc7633402d089f
I20260925 01:28:28.030953 29081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:28.041674 29081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4: Bootstrap starting.
I20260925 01:28:28.044592 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:28.030203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57481 (local address 127.6.236.190:33875)
0925 01:28:28.031013 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:28.031028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:28.031075 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:28:28.033247 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:28.033271 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:28.033679 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:28.034120 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:28.042663 (+ 8543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:28.042672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:28.042688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:28.042708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:28.042820 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:28.042849 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:28.043099 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:28.043242 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 01:28:28.043448 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":299,"threads_started":1}
I20260925 01:28:28.045215 29081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:28.046214 29081 log.cc:824] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:28.048316 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:28.049172 29081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4: No bootstrap required, opened a new log
I20260925 01:28:28.055795 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:28.053424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37113 (local address 127.6.236.190:33875)
0925 01:28:28.053609 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:28.053616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:28.053897 (+ 281us) server_negotiation.cc:407] Connection header received
0925 01:28:28.054041 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:28.054046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:28.054117 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:28.054223 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:28.055179 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:28.055185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:28.055190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:28.055209 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:28.055261 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:28.055281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:28.055403 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:28.055451 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 01:28:28.055506 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:28:28.058696 29081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } }
I20260925 01:28:28.059036 29081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:28.059126 29081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd7fe49eb49458d94b908dd099a76e4, State: Initialized, Role: FOLLOWER
I20260925 01:28:28.059695 29081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } }
I20260925 01:28:28.059911 29081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:28.060039 29081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:28.060199 29081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:28.062346 29081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } }
I20260925 01:28:28.062808 29081 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd7fe49eb49458d94b908dd099a76e4; no voters:
I20260925 01:28:28.063436 29081 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:28.063526 29091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:28.064124 29091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd7fe49eb49458d94b908dd099a76e4, State: Running, Role: LEADER
I20260925 01:28:28.064653 29091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } }
I20260925 01:28:28.065393 29081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:28.067322 29092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd7fe49eb49458d94b908dd099a76e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } } }
I20260925 01:28:28.067472 29093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd7fe49eb49458d94b908dd099a76e4. Latest consensus state: current_term: 1 leader_uuid: "6fd7fe49eb49458d94b908dd099a76e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7fe49eb49458d94b908dd099a76e4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33875 } } }
I20260925 01:28:28.067745 29093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:28.067746 29092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:28.070214 29099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:28.075201 29099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:28.083966 29099 catalog_manager.cc:1347] Generated new cluster ID: e63e2f488dac43e3a630ecc45ad83ae5
I20260925 01:28:28.084064 29099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:28.101146 29099 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:28.101536 29099 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:28.111850 29099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd7fe49eb49458d94b908dd099a76e4: Generated new TSK 0
I20260925 01:28:28.112591 29099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:28.119304 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29014
W20260925 01:28:28.120561 29076 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57481
W20260925 01:28:28.121160 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:28.121307 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29014
2026-09-25T01:28:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1751 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:28 dist-test-slave-s0b2 krb5kdc[29112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:28 dist-test-slave-s0b2 krb5kdc[29112](info): set up 2 sockets
Sep 25 01:28:28 dist-test-slave-s0b2 krb5kdc[29112](info): commencing operation
krb5kdc: starting...
W20260925 01:28:29.147280 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:29 dist-test-slave-s0b2 krb5kdc[29112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:29.385196 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42735
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:42735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:29.701556 29125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:29.701875 29125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:29.702008 29125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:29.709348 29125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:29.709467 29125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:29.709544 29125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:29.709618 29125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:29.709677 29125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:29.719677 29125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:42735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:29.720979 29125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:29.722321 29125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:29.733376 29133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:29.733397 29131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:29.733397 29130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:29.734169 29125 server_base.cc:1034] running on GCE node
I20260925 01:28:29.734915 29125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:29.736315 29125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:29.737505 29125 hybrid_clock.cc:648] HybridClock initialized: now 1790299709737450 us; error 45 us; skew 500 ppm
I20260925 01:28:29.737812 29125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:29 dist-test-slave-s0b2 krb5kdc[29112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:29.749922 29125 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:29.751605 29125 webserver.cc:460] Webserver started at http://127.6.236.190:38927/ using document root <none> and password file <none>
I20260925 01:28:29.752287 29125 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:29.752391 29125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:29.752656 29125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:29.755764 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "8d0d2641e53749b68502b588d1da7287"
format_stamp: "Formatted at 2026-09-25 01:28:29 on dist-test-slave-s0b2"
server_key: "502810ac0aa244464dedd0460ae355fb"
server_key_iv: "30f3b6afdd7cf3637bf12d99662ea3db"
server_key_version: "encryptionkey@0"
I20260925 01:28:29.756541 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "8d0d2641e53749b68502b588d1da7287"
format_stamp: "Formatted at 2026-09-25 01:28:29 on dist-test-slave-s0b2"
server_key: "502810ac0aa244464dedd0460ae355fb"
server_key_iv: "30f3b6afdd7cf3637bf12d99662ea3db"
server_key_version: "encryptionkey@0"
I20260925 01:28:29.761736 29125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 01:28:29.765285 29140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:29.766671 29125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:29.766825 29125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "8d0d2641e53749b68502b588d1da7287"
format_stamp: "Formatted at 2026-09-25 01:28:29 on dist-test-slave-s0b2"
server_key: "502810ac0aa244464dedd0460ae355fb"
server_key_iv: "30f3b6afdd7cf3637bf12d99662ea3db"
server_key_version: "encryptionkey@0"
I20260925 01:28:29.766992 29125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:29.795248 29125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:29.799666 29125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:29.799968 29125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:29.818222 29125 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:42735
I20260925 01:28:29.818275 29191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:42735 every 8 connection(s)
I20260925 01:28:29.819603 29125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:29.823416 29192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:29.827390 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29125
I20260925 01:28:29.827543 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:29.827982 7090 external_mini_cluster.cc:1428] Setting key 7a023a8620886e6c67c7fa6c20c97fd1
I20260925 01:28:29.837046 29192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287: Bootstrap starting.
I20260925 01:28:29.840873 29192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:29.842082 29192 log.cc:824] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:28:29 dist-test-slave-s0b2 krb5kdc[29112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:28:29.845000 29192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287: No bootstrap required, opened a new log
I20260925 01:28:29.851378 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:29.830035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34311 (local address 127.6.236.190:42735)
0925 01:28:29.830843 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:29.830858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:29.830910 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:28:29.832691 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:29.832730 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:29.833309 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:29.833839 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:29.835179 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:29.837126 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:29.838538 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:29.839609 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:29.839796 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:29.844073 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:29.844081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:29.844097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:29.844134 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:29.847276 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:29.847972 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:29.847978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:29.847984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:29.848156 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:29.848537 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:29.848541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:29.848542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:29.849091 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:29.849388 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:29.849870 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 01:28:29.850331 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
I20260925 01:28:29.852918 29192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } }
I20260925 01:28:29.853291 29192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:29.853402 29192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0d2641e53749b68502b588d1da7287, State: Initialized, Role: FOLLOWER
I20260925 01:28:29.854056 29192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } }
I20260925 01:28:29.854316 29192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:29.854458 29192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:29.854610 29192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:29.854991 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:29.857630 29192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } }
I20260925 01:28:29.858220 29192 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0d2641e53749b68502b588d1da7287; no voters:
I20260925 01:28:29.858932 29192 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:29.859066 29199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:29.859689 29199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0d2641e53749b68502b588d1da7287, State: Running, Role: LEADER
I20260925 01:28:29.861302 29199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } }
I20260925 01:28:29.861759 29192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:29.864905 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0d2641e53749b68502b588d1da7287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } } }
I20260925 01:28:29.865222 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:29.864821 29202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0d2641e53749b68502b588d1da7287. Latest consensus state: current_term: 1 leader_uuid: "8d0d2641e53749b68502b588d1da7287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0d2641e53749b68502b588d1da7287" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 42735 } } }
I20260925 01:28:29.867060 29202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:29.872786 29209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:29.878121 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:29.860432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49545 (local address 127.6.236.190:42735)
0925 01:28:29.860611 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:29.860617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:29.861051 (+ 434us) server_negotiation.cc:407] Connection header received
0925 01:28:29.861245 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:29.861250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:29.861333 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:29.861463 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:29.863312 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:29.864556 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:29.867338 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:29.868066 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:29.868207 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:29.871002 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:29.871009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:29.871032 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:29.871077 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:29.875065 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:29.875860 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:29.875865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:29.875869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:29.876036 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:29.876420 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:29.876429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:29.876433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:29.876890 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:29.877168 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:29.877526 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 01:28:29.877767 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:28:29.878793 29209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:29.888715 29209 catalog_manager.cc:1347] Generated new cluster ID: d3dd26191749490bae841385f9347222
I20260925 01:28:29.888856 29209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:29.912582 29209 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:29.913885 29209 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:29.929594 29209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0d2641e53749b68502b588d1da7287: Generated new TSK 0
I20260925 01:28:29.930471 29209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:29.943903 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29125
I20260925 01:28:29.945386 29187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:30.037613 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29125
2026-09-25T01:28:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:30 dist-test-slave-s0b2 krb5kdc[29224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:30 dist-test-slave-s0b2 krb5kdc[29224](info): set up 2 sockets
Sep 25 01:28:30 dist-test-slave-s0b2 krb5kdc[29224](info): commencing operation
krb5kdc: starting...
W20260925 01:28:31.043236 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:31 dist-test-slave-s0b2 krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:31.275417 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46637
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:46637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:31.587739 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:31.588009 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:31.588107 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:31.595099 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:31.595197 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:31.595249 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:31.595307 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:31.595350 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:31.604856 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:46637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:46637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:31.606068 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:31.607383 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:31.618319 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:31.618332 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:31.618541 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:31.619112 29237 server_base.cc:1034] running on GCE node
I20260925 01:28:31.619782 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:31.621125 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:31.622327 29237 hybrid_clock.cc:648] HybridClock initialized: now 1790299711622271 us; error 45 us; skew 500 ppm
I20260925 01:28:31.622643 29237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:31 dist-test-slave-s0b2 krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299711, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:31.625523 29237 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:31.626958 29237 webserver.cc:460] Webserver started at http://127.6.236.190:38799/ using document root <none> and password file <none>
I20260925 01:28:31.627571 29237 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:31.627691 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:31.627969 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:31.630844 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "0d1e8546fa9945868ebdfda2a54eb3ba"
format_stamp: "Formatted at 2026-09-25 01:28:31 on dist-test-slave-s0b2"
server_key: "cb82b458a8ece704731333d23fe4d4ad"
server_key_iv: "05f115202b7b7ad8e75c612c9cef8920"
server_key_version: "encryptionkey@0"
I20260925 01:28:31.631541 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "0d1e8546fa9945868ebdfda2a54eb3ba"
format_stamp: "Formatted at 2026-09-25 01:28:31 on dist-test-slave-s0b2"
server_key: "cb82b458a8ece704731333d23fe4d4ad"
server_key_iv: "05f115202b7b7ad8e75c612c9cef8920"
server_key_version: "encryptionkey@0"
I20260925 01:28:31.636756 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:28:31.640051 29252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:31.641292 29237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:28:31.641438 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "0d1e8546fa9945868ebdfda2a54eb3ba"
format_stamp: "Formatted at 2026-09-25 01:28:31 on dist-test-slave-s0b2"
server_key: "cb82b458a8ece704731333d23fe4d4ad"
server_key_iv: "05f115202b7b7ad8e75c612c9cef8920"
server_key_version: "encryptionkey@0"
I20260925 01:28:31.641646 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:31.658175 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:31.662039 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:31.662325 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:31.678960 29237 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:46637
I20260925 01:28:31.678983 29303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:46637 every 8 connection(s)
I20260925 01:28:31.680284 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:31.683946 29304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:31.685738 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29237
I20260925 01:28:31.685911 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:31.686463 7090 external_mini_cluster.cc:1428] Setting key e1a89e7282c6cd2e593919f815cefe87
I20260925 01:28:31.697343 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba: Bootstrap starting.
I20260925 01:28:31.699136 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:31.688197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49185 (local address 127.6.236.190:46637)
0925 01:28:31.688995 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:31.689007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:31.689050 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:28:31.690637 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:31.690659 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:31.691048 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:31.691382 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:31.691911 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:31.693170 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:31.694989 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:31.695645 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:31.695808 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:31.697001 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:31.697008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:31.697021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:31.697547 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:31.697626 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:31.697654 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:31.697751 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:31.697960 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 01:28:31.698196 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":387,"threads_started":1}
I20260925 01:28:31.700519 29304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:31.701627 29304 log.cc:824] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:31.702533 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:31.704193 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba: No bootstrap required, opened a new log
I20260925 01:28:31.712797 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:31.706487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52323 (local address 127.6.236.190:46637)
0925 01:28:31.706697 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:31.706704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:31.707178 (+ 474us) server_negotiation.cc:407] Connection header received
0925 01:28:31.707327 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:31.707331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:31.707393 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:31.707470 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:31.708009 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:31.708720 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:31.710549 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:31.711008 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:31.711107 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:31.712159 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:31.712164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:31.712167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:31.712185 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:31.712238 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:31.712254 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:31.712323 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:31.712460 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 01:28:31.712527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:28:31.713299 29304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } }
I20260925 01:28:31.713646 29304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:31.713740 29304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1e8546fa9945868ebdfda2a54eb3ba, State: Initialized, Role: FOLLOWER
I20260925 01:28:31.714382 29304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } }
I20260925 01:28:31.714607 29304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:31.714733 29304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:31.714880 29304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:31.717267 29304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } }
I20260925 01:28:31.717717 29304 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1e8546fa9945868ebdfda2a54eb3ba; no voters:
I20260925 01:28:31.718211 29304 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:31.718343 29314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:31.719419 29314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [term 1 LEADER]: Becoming Leader. State: Replica: 0d1e8546fa9945868ebdfda2a54eb3ba, State: Running, Role: LEADER
I20260925 01:28:31.719929 29314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } }
I20260925 01:28:31.720176 29304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:31.722615 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } } }
I20260925 01:28:31.722556 29316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1e8546fa9945868ebdfda2a54eb3ba. Latest consensus state: current_term: 1 leader_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e8546fa9945868ebdfda2a54eb3ba" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 46637 } } }
I20260925 01:28:31.723007 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:31.723023 29316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:31.725656 29322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:31.730415 29322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:31.737946 29322 catalog_manager.cc:1347] Generated new cluster ID: 1afb96c18c514eedab0a3fd6665e7032
I20260925 01:28:31.738042 29322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:31.764394 29322 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:31.766305 29322 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:31.782456 29322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d1e8546fa9945868ebdfda2a54eb3ba: Generated new TSK 0
I20260925 01:28:31.783465 29322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:31.800180 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29237
W20260925 01:28:31.801512 29299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49185
W20260925 01:28:31.802213 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:31.802374 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29237
2026-09-25T01:28:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1764 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:31 dist-test-slave-s0b2 krb5kdc[29335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:31 dist-test-slave-s0b2 krb5kdc[29335](info): set up 2 sockets
Sep 25 01:28:31 dist-test-slave-s0b2 krb5kdc[29335](info): commencing operation
krb5kdc: starting...
W20260925 01:28:32.815347 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:32 dist-test-slave-s0b2 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:33.050843 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:39943
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:33.369006 29348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:33.369308 29348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:33.369436 29348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:33.376637 29348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:33.376766 29348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:33.376839 29348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:33.376917 29348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:33.376978 29348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:33.387003 29348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:39943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:33.388321 29348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:33.389605 29348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:33.400627 29354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:33.400646 29356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:33.400713 29353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:33.401367 29348 server_base.cc:1034] running on GCE node
I20260925 01:28:33.402077 29348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:33.403458 29348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:33.404688 29348 hybrid_clock.cc:648] HybridClock initialized: now 1790299713404639 us; error 27 us; skew 500 ppm
I20260925 01:28:33.405042 29348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:33 dist-test-slave-s0b2 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299713, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:33.408090 29348 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:33.409416 29348 webserver.cc:460] Webserver started at http://127.6.236.190:39221/ using document root <none> and password file <none>
I20260925 01:28:33.410073 29348 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:33.410190 29348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:33.410499 29348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:33.413394 29348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "31f1fdbc074b465c955eb5c974cd8ba4"
format_stamp: "Formatted at 2026-09-25 01:28:33 on dist-test-slave-s0b2"
server_key: "cab25dc25b4f10e357b6a54205990c07"
server_key_iv: "885234dff65ff7f20e6a8887f38e70bc"
server_key_version: "encryptionkey@0"
I20260925 01:28:33.414137 29348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "31f1fdbc074b465c955eb5c974cd8ba4"
format_stamp: "Formatted at 2026-09-25 01:28:33 on dist-test-slave-s0b2"
server_key: "cab25dc25b4f10e357b6a54205990c07"
server_key_iv: "885234dff65ff7f20e6a8887f38e70bc"
server_key_version: "encryptionkey@0"
I20260925 01:28:33.419166 29348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:28:33.422554 29363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:33.423878 29348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:28:33.424028 29348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "31f1fdbc074b465c955eb5c974cd8ba4"
format_stamp: "Formatted at 2026-09-25 01:28:33 on dist-test-slave-s0b2"
server_key: "cab25dc25b4f10e357b6a54205990c07"
server_key_iv: "885234dff65ff7f20e6a8887f38e70bc"
server_key_version: "encryptionkey@0"
I20260925 01:28:33.424206 29348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:33.429895 29348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:33.433808 29348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:33.434091 29348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:33.450088 29348 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:39943
I20260925 01:28:33.450109 29414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:39943 every 8 connection(s)
I20260925 01:28:33.451443 29348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:33.452986 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29348
I20260925 01:28:33.453172 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:33.453621 7090 external_mini_cluster.cc:1428] Setting key e09877e871653ac97d9c8f682fb3262d
I20260925 01:28:33.455425 29415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:33.461097 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:33.455402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58715 (local address 127.6.236.190:39943)
0925 01:28:33.456152 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:33.456164 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:33.456205 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:28:33.457768 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:33.457790 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:33.458144 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:33.458556 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:33.459520 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:33.459529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:33.459542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:33.459556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:33.459640 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:33.459660 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:33.459874 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:33.460015 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 01:28:33.460209 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":299,"threads_started":1}
I20260925 01:28:33.464565 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:33.466665 29415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4: Bootstrap starting.
I20260925 01:28:33.470042 29415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:33.471302 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:33.468771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57155 (local address 127.6.236.190:39943)
0925 01:28:33.468904 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:33.468908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:33.469343 (+ 435us) server_negotiation.cc:407] Connection header received
0925 01:28:33.469493 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:33.469497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:33.469542 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:33.469628 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:33.470598 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:33.470604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:33.470608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:28:33.470626 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:33.470675 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:28:33.470695 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:33.470810 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:33.470957 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 01:28:33.471022 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 01:28:33.471334 29415 log.cc:824] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:33.474015 29415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4: No bootstrap required, opened a new log
I20260925 01:28:33.481448 29415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } }
I20260925 01:28:33.481822 29415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:33.481899 29415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f1fdbc074b465c955eb5c974cd8ba4, State: Initialized, Role: FOLLOWER
I20260925 01:28:33.482404 29415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } }
I20260925 01:28:33.482606 29415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:33.482697 29415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:33.482821 29415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:33.485044 29415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } }
I20260925 01:28:33.485469 29415 leader_election.cc:302] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f1fdbc074b465c955eb5c974cd8ba4; no voters:
I20260925 01:28:33.485985 29415 leader_election.cc:288] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:33.486117 29425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:33.486720 29425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 31f1fdbc074b465c955eb5c974cd8ba4, State: Running, Role: LEADER
I20260925 01:28:33.487241 29425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } }
I20260925 01:28:33.487879 29415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:33.489885 29427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f1fdbc074b465c955eb5c974cd8ba4. Latest consensus state: current_term: 1 leader_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } } }
I20260925 01:28:33.490302 29427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:33.491490 29431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:33.491465 29426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f1fdbc074b465c955eb5c974cd8ba4" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 39943 } } }
I20260925 01:28:33.491758 29426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:33.496294 29431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:33.516500 29431 catalog_manager.cc:1347] Generated new cluster ID: 6fe1aaf9ed5e44f5b17dab73e04b5d3b
I20260925 01:28:33.516614 29431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:33.542030 29431 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:33.542557 29431 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:33.553212 29431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f1fdbc074b465c955eb5c974cd8ba4: Generated new TSK 0
I20260925 01:28:33.554224 29431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:33.573025 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29348
W20260925 01:28:33.574179 29410 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58715
W20260925 01:28:33.574925 7090 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:28:33.575062 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29348
2026-09-25T01:28:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1772 ms)
[----------] 18 tests from AuthTokenIssuingTest (32381 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:33 dist-test-slave-s0b2 krb5kdc[29446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:33 dist-test-slave-s0b2 krb5kdc[29446](info): set up 2 sockets
Sep 25 01:28:33 dist-test-slave-s0b2 krb5kdc[29446](info): commencing operation
krb5kdc: starting...
W20260925 01:28:34.599454 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:34 dist-test-slave-s0b2 krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:34.840929 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37263
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:35.154939 29459 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:35.155258 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:35.155359 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:35.155473 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:35.162575 29459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:35.162658 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:35.162727 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:35.162792 29459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:35.162876 29459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:35.172567 29459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:35.173856 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:35.175206 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:35.186192 29467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:35.186399 29465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:35.186765 29464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:35.187050 29459 server_base.cc:1034] running on GCE node
I20260925 01:28:35.187989 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:35.189729 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:35.190961 29459 hybrid_clock.cc:648] HybridClock initialized: now 1790299715190842 us; error 71 us; skew 500 ppm
I20260925 01:28:35.191359 29459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:35.200773 29459 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:35.202169 29459 webserver.cc:460] Webserver started at http://127.6.236.190:46289/ using document root <none> and password file <none>
I20260925 01:28:35.202888 29459 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:35.202983 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:35.203280 29459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:35.206169 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "f1117ff989d1419892c220fb25297f11"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "16fbe4df041ba8654660d8ed245f716e"
server_key_iv: "bfbd84b1a1f158e57e1be52dff0932ba"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.206975 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "f1117ff989d1419892c220fb25297f11"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "16fbe4df041ba8654660d8ed245f716e"
server_key_iv: "bfbd84b1a1f158e57e1be52dff0932ba"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.212051 29459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:28:35.215288 29474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:35.216526 29459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:35.216683 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "f1117ff989d1419892c220fb25297f11"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "16fbe4df041ba8654660d8ed245f716e"
server_key_iv: "bfbd84b1a1f158e57e1be52dff0932ba"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.216869 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:35.244347 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:35.248073 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:35.248319 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:35.264993 29459 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:37263
I20260925 01:28:35.265000 29525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:37263 every 8 connection(s)
I20260925 01:28:35.266273 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:35.269830 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:35.272882 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29459
I20260925 01:28:35.273067 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:35.273690 7090 external_mini_cluster.cc:1428] Setting key 3cd1cef52e31824f6c4af2c70e755b44
I20260925 01:28:35.276233 29526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.189@KRBTEST.COM; defaulting to no policy
I20260925 01:28:35.298660 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.279303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37263 (local address 127.0.0.1:54934)
0925 01:28:35.280311 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.280345 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:35.282234 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:35.283104 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:35.283114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:35.283806 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:35.284228 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.284255 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.285794 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.285799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.287330 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.287344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.288263 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.288270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.288436 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.289078 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:35.289103 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:35.291594 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:35.294595 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.294608 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.294622 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.295163 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.295532 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.295537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.295541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.295751 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.296518 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:35.296526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:35.296804 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 01:28:35.297078 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 01:28:35.297661 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":88,"thread_start_us":640,"threads_started":1}
I20260925 01:28:35.298678 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.279302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.6.236.190:37263)
0925 01:28:35.280030 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:35.280045 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:35.280460 (+ 415us) server_negotiation.cc:407] Connection header received
0925 01:28:35.282650 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:35.282656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:35.282878 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:35.283028 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:35.284446 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.285646 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.287471 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.288026 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.288168 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.291808 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:35.291816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:35.291828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:35.291857 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:35.294470 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.295282 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.295288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.295290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.295428 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.295875 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.295879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.295881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.296374 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:35.296526 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:35.297205 (+ 679us) server_negotiation.cc:299] Negotiation successful
0925 01:28:35.297491 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":511,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.189@KRBTEST.COM" created.
I20260925 01:28:35.301527 29526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:35.302675 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.302276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35498)
0925 01:28:35.302435 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.302546 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:28:35.302927 29477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
I20260925 01:28:35.304382 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.304042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35504)
0925 01:28:35.304192 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.304257 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:28:35.304566 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:44009: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
I20260925 01:28:35.329123 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:44009
--webserver_interface=127.6.236.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:28:35.353874 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } attempt: 1
I20260925 01:28:35.354969 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.354522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35510)
0925 01:28:35.354713 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.354838 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 01:28:35.355967 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.355633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35514)
0925 01:28:35.355774 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.355845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:28:35.356135 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:44009: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
I20260925 01:28:35.424368 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } attempt: 2
I20260925 01:28:35.425458 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.424973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35518)
0925 01:28:35.425203 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.425320 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:28:35.426457 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.426052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35524)
0925 01:28:35.426206 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.426324 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 01:28:35.426625 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:44009: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
I20260925 01:28:35.517863 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } attempt: 3
I20260925 01:28:35.518947 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.518528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35538)
0925 01:28:35.518718 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.518818 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:28:35.520049 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.519685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35546)
0925 01:28:35.519825 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.519896 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 01:28:35.520239 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:44009: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
W20260925 01:28:35.648090 29533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:35.648397 29533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:35.648469 29533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:35.648550 29533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:35.655757 29533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:35.655845 29533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:35.655906 29533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:35.655956 29533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:35.656014 29533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:35.663465 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } attempt: 4
I20260925 01:28:35.664686 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.664140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35554)
0925 01:28:35.664463 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.664551 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 01:28:35.665688 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.665382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35564)
0925 01:28:35.665495 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.665575 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 01:28:35.665920 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:44009: Network error: Client connection negotiation failed: client connection to 127.6.236.189:44009: connect: Connection refused (error 111)
I20260925 01:28:35.666471 29533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:44009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.236.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:35.667706 29533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:35.668985 29533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:35.680199 29538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:35.680171 29541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:35.680140 29539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:35.680867 29533 server_base.cc:1034] running on GCE node
I20260925 01:28:35.681666 29533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:35.683039 29533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:35.684254 29533 hybrid_clock.cc:648] HybridClock initialized: now 1790299715684189 us; error 43 us; skew 500 ppm
I20260925 01:28:35.684556 29533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:35.687498 29533 init.cc:377] Logged in from keytab as kudu/127.6.236.189@KRBTEST.COM (short username kudu)
I20260925 01:28:35.688907 29533 webserver.cc:460] Webserver started at http://127.6.236.189:37505/ using document root <none> and password file <none>
I20260925 01:28:35.689558 29533 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:35.689657 29533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:35.689957 29533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:35.692919 29533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data/instance:
uuid: "83b128306b064a36bfb2a96786c88ae7"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "4736971d28a4906a8f754db0ea126003"
server_key_iv: "9bee2a63223d10ca54ddb93ea341f39b"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.693681 29533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal/instance:
uuid: "83b128306b064a36bfb2a96786c88ae7"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "4736971d28a4906a8f754db0ea126003"
server_key_iv: "9bee2a63223d10ca54ddb93ea341f39b"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.698625 29533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:28:35.701949 29548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:35.703341 29533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:35.703496 29533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "83b128306b064a36bfb2a96786c88ae7"
format_stamp: "Formatted at 2026-09-25 01:28:35 on dist-test-slave-s0b2"
server_key: "4736971d28a4906a8f754db0ea126003"
server_key_iv: "9bee2a63223d10ca54ddb93ea341f39b"
server_key_version: "encryptionkey@0"
I20260925 01:28:35.703694 29533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:35.740983 29533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:35.744829 29533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:35.745065 29533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:35.761781 29533 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.189:44009
I20260925 01:28:35.761801 29599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.189:44009 every 8 connection(s)
I20260925 01:28:35.763132 29533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
I20260925 01:28:35.766717 29600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:35.772850 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29533
I20260925 01:28:35.773006 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-1/wal/instance
I20260925 01:28:35.773408 7090 external_mini_cluster.cc:1428] Setting key 6d1cbd37028eba40a55f679ac0384a29
I20260925 01:28:35.772976 29600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:28:35.792171 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.775576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.6.236.190:37263)
0925 01:28:35.775743 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:35.775748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:35.776957 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 01:28:35.778801 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:35.778807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:35.778890 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:35.778981 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:35.780447 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.781133 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.782705 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.783130 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.783255 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.786714 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:35.786719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:35.786723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:35.786761 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:35.789184 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.789997 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.790001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.790003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.790129 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.790550 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.790553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.790556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.790830 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:35.790949 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:35.791710 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 01:28:35.791882 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:28:35.793686 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.776105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37263 (local address 127.0.0.1:54948)
0925 01:28:35.776797 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.776839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:35.778532 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:35.779217 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:35.779225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:35.779719 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:35.780275 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.780293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.781286 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.781290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.782559 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.782572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.783347 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.783354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.783582 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.784180 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:35.784207 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:35.786496 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:35.789317 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.789325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.789341 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.789882 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.790220 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.790224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.790227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.790454 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.790945 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:35.790952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:35.791369 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 01:28:35.791650 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 01:28:35.792141 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":271,"threads_started":1}
I20260925 01:28:35.795382 29600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.236.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.188@KRBTEST.COM" created.
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:35.809163 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.796080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35570)
0925 01:28:35.796300 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.796319 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:35.796420 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:35.797387 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:35.797390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:35.797762 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:35.797901 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.797910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.799408 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.799413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.800237 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.800248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.801003 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.801008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.801177 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.801687 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:35.801703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:35.803719 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:35.806661 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.806666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.806671 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.807111 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.807471 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.807476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.807480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.807619 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.808243 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:35.808248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:35.808382 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 01:28:35.808505 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 01:28:35.808783 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 01:28:35.809266 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.796314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.6.236.189:44009)
0925 01:28:35.796918 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:35.796924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:35.796954 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:28:35.796997 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:35.797004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:35.797235 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:35.797349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:35.798041 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.799284 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.800359 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.800874 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.800979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.803919 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:35.803930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:35.803942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:35.803974 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:35.806550 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.807232 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.807238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.807241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.807375 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.807712 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.807718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.807722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.808134 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:35.808324 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:35.808655 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:28:35.808900 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":466,"thread_start_us":318,"threads_started":1}
I20260925 01:28:35.811957 29600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:35.813143 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.812627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37896)
0925 01:28:35.812823 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.812968 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:28:35.813407 29552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:35.815007 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.814655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37902)
0925 01:28:35.814799 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.814890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 01:28:35.815217 29600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
I20260925 01:28:35.827596 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:45261
--webserver_interface=127.6.236.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:28:35.864539 29600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 1
I20260925 01:28:35.865788 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.865263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37912)
0925 01:28:35.865466 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.865597 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:28:35.866780 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.866404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37926)
0925 01:28:35.866556 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.866645 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 01:28:35.866973 29600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:35.935181 29600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 2
I20260925 01:28:35.936354 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.935839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37936)
0925 01:28:35.936049 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.936162 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:28:35.937306 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.936945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37942)
0925 01:28:35.937094 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.937173 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:28:35.937523 29600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:35.965137 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } attempt: 5
Sep 25 01:28:35 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:35.978207 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.965929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.6.236.189:44009)
0925 01:28:35.966114 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:35.966120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:35.966193 (+ 73us) server_negotiation.cc:407] Connection header received
0925 01:28:35.966434 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:35.966440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:35.966505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:35.966583 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:35.967369 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.968061 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.969192 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.969710 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.969841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.973000 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:35.973005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:35.973008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:35.973041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:35.975723 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.976511 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.976514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.976516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.976629 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:35.977019 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:35.977022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:35.977024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:35.977283 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:35.977390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:35.977834 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 01:28:35.977992 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:28:35.978428 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.965878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35576)
0925 01:28:35.966090 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.966112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:35.966292 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:35.966637 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:35.966643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:35.967025 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:35.967205 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.967215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.968191 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.968196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.969047 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:35.969058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:35.969864 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:35.969869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:35.970061 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:35.970656 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:35.970674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:35.972812 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:35.975865 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.975871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.975875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.976405 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.976732 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:35.976737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:35.976741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:35.976920 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:35.977395 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:35.977400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:35.977577 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 01:28:35.977752 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 01:28:35.978057 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:28:35.979415 29526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:35.980482 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.980019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37952)
0925 01:28:35.980211 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.980311 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:28:35.981446 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:35.981056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37960)
0925 01:28:35.981222 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:35.981315 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:28:35.981709 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:36.028717 29600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 3
I20260925 01:28:36.030015 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.029385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37966)
0925 01:28:36.029653 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.029795 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 01:28:36.031196 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.030705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37972)
0925 01:28:36.030948 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.031054 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260925 01:28:36.031414 29600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:36.032964 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 1
I20260925 01:28:36.033926 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.033473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37976)
0925 01:28:36.033656 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.033756 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:28:36.034862 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.034520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:37984)
0925 01:28:36.034654 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.034742 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 01:28:36.035094 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:36.103317 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 2
I20260925 01:28:36.104521 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.103979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38000)
0925 01:28:36.104218 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.104331 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 01:28:36.105501 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.105135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38004)
0925 01:28:36.105279 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.105363 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:28:36.105710 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
W20260925 01:28:36.145721 29607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:36.146011 29607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:36.146111 29607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:36.146229 29607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:36.153302 29607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:36.153388 29607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:36.153476 29607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:36.153534 29607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:36.153622 29607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:36.163419 29607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:37263,127.6.236.189:44009,127.6.236.188:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.236.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:36.164657 29607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:36.165926 29607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:28:36.174611 29600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 4
I20260925 01:28:36.175724 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.175271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38010)
0925 01:28:36.175473 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.175558 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 01:28:36.176893 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.176461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38018)
0925 01:28:36.176626 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.176760 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 01:28:36.177145 29600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
W20260925 01:28:36.177183 29613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:36.177531 29615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:36.177809 29612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:36.178216 29607 server_base.cc:1034] running on GCE node
I20260925 01:28:36.178958 29607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:36.180272 29607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:36.181483 29607 hybrid_clock.cc:648] HybridClock initialized: now 1790299716181425 us; error 30 us; skew 500 ppm
I20260925 01:28:36.181815 29607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:36.184716 29607 init.cc:377] Logged in from keytab as kudu/127.6.236.188@KRBTEST.COM (short username kudu)
I20260925 01:28:36.186084 29607 webserver.cc:460] Webserver started at http://127.6.236.188:43309/ using document root <none> and password file <none>
I20260925 01:28:36.186790 29607 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:36.186890 29607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:36.187191 29607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:36.190034 29607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data/instance:
uuid: "94d12fc306e0479dbfb41ae632a1ccf4"
format_stamp: "Formatted at 2026-09-25 01:28:36 on dist-test-slave-s0b2"
server_key: "ed46e3fe6c954dcd77db10f2596976f6"
server_key_iv: "ff01cae03997e4e64d9372c583b663f1"
server_key_version: "encryptionkey@0"
I20260925 01:28:36.190832 29607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal/instance:
uuid: "94d12fc306e0479dbfb41ae632a1ccf4"
format_stamp: "Formatted at 2026-09-25 01:28:36 on dist-test-slave-s0b2"
server_key: "ed46e3fe6c954dcd77db10f2596976f6"
server_key_iv: "ff01cae03997e4e64d9372c583b663f1"
server_key_version: "encryptionkey@0"
I20260925 01:28:36.195591 29607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 01:28:36.198825 29622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:36.200071 29607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:28:36.200219 29607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "94d12fc306e0479dbfb41ae632a1ccf4"
format_stamp: "Formatted at 2026-09-25 01:28:36 on dist-test-slave-s0b2"
server_key: "ed46e3fe6c954dcd77db10f2596976f6"
server_key_iv: "ff01cae03997e4e64d9372c583b663f1"
server_key_version: "encryptionkey@0"
I20260925 01:28:36.200407 29607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:36.211905 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 3
I20260925 01:28:36.212916 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.212504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38020)
0925 01:28:36.212672 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.212765 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:28:36.213814 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.213468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38028)
0925 01:28:36.213637 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.213710 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 01:28:36.213996 29526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:45261: Network error: Client connection negotiation failed: client connection to 127.6.236.188:45261: connect: Connection refused (error 111)
I20260925 01:28:36.221213 29607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:36.225135 29607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:36.225373 29607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:36.240634 29607 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.188:45261
I20260925 01:28:36.240644 29673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.188:45261 every 8 connection(s)
I20260925 01:28:36.241864 29607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
I20260925 01:28:36.245510 29674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:36.249758 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29607
I20260925 01:28:36.249913 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299375376939-7090-0/minicluster-data/master-2/wal/instance
I20260925 01:28:36.250360 7090 external_mini_cluster.cc:1428] Setting key c76cc9d446bf67e75df13ad873435cdc
I20260925 01:28:36.252280 29674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:28:36.270785 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.254684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.6.236.190:37263)
0925 01:28:36.254853 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.254859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.255355 (+ 496us) server_negotiation.cc:407] Connection header received
0925 01:28:36.255548 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.255553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.255633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.255704 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.257424 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.258115 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.260158 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.260588 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.260685 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.264145 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.264150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.264153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.264188 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.267606 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.268444 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.268449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.268452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.268612 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.269071 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.269076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.269079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.269390 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.269480 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.270146 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.270394 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:28:36.271955 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.254655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:37263 (local address 127.0.0.1:54958)
0925 01:28:36.255207 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.255247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:36.255396 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:36.255965 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:36.255972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:36.256747 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:36.257252 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.257272 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.258294 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.258299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.260020 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.260035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.260810 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.260817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.260997 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.261767 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:36.261787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:36.263965 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:36.267760 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.267772 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.267782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.268317 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.268719 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.268724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.268735 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.268938 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.269563 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:36.269568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:36.269744 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 01:28:36.271064 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0925 01:28:36.271290 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":4,"thread_start_us":231,"threads_started":1}
I20260925 01:28:36.271981 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.252118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.6.236.188:45261)
0925 01:28:36.252859 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.252870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.252911 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:28:36.254693 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.254719 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.255225 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.255537 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.257267 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.258440 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.260197 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.260774 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.260950 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.264740 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.264751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.264765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.264800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.267936 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.268550 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.268558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.268562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.268744 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.269137 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.269141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.269145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.269769 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.270050 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.270403 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.270776 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":516,"thread_start_us":330,"threads_started":1}
I20260925 01:28:36.273707 29674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:28:36.286458 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.274756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:44009 (local address 127.0.0.1:35586)
0925 01:28:36.274991 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.275006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:36.275099 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:36.275425 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:36.275429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:36.275825 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:36.275968 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.275976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.276897 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.276903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.277758 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.277769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.278468 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.278474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.278668 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.279202 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:36.279224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:36.281387 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:36.284242 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.284246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.284249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.284694 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.285034 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.285038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.285042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.285222 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.285674 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:36.285679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:36.285835 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 01:28:36.285943 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:28:36.286126 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 01:28:36.286618 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.274481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.6.236.189:44009)
0925 01:28:36.274653 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.274660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.275088 (+ 428us) server_negotiation.cc:407] Connection header received
0925 01:28:36.275230 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.275235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.275298 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.275376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.276101 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.276770 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.277886 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.278330 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.278435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.281590 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.281596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.281600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.281642 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.284115 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.284798 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.284802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.284804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.284933 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.285315 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.285318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.285320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.285569 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.285675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.286076 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.286307 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:28:36.287740 29674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:36.288494 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.275014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.6.236.190:37263)
0925 01:28:36.275228 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.275232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.275247 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:28:36.275292 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.275297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.275359 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.275453 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.276586 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.277308 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.278445 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.278869 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.278986 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.282976 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.282982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.282987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.283023 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.285675 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.286393 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.286397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.286401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.286551 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.286934 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.286940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.286943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.287238 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.287472 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.288029 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.288224 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:36.300457 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.288829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.6.236.188:45261)
0925 01:28:36.288953 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.288958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.288972 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:28:36.289010 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.289015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.289072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.289147 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.289833 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.290651 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.291690 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.292063 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.292163 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.295015 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.295022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.295026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.295060 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.298026 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.298810 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.298813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.298816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.298932 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.299247 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.299252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.299255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.299531 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.299664 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.300047 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.300242 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":30}
I20260925 01:28:36.300782 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.288483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38038)
0925 01:28:36.288648 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.288667 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:36.288756 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:36.289180 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:36.289184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:36.289592 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:36.289718 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.289726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.290760 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.290765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.291575 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.291585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.292182 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.292188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.292378 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.292865 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:36.292880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:36.294843 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:36.298159 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.298164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.298167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.298709 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.299030 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.299035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.299038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.299156 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.299663 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:36.299668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:36.299815 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 01:28:36.299982 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:28:36.300388 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":122}
I20260925 01:28:36.304252 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.290158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.6.236.189:44009)
0925 01:28:36.290323 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.290328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.290344 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:28:36.290485 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.290490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.290549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.290650 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.291730 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.292409 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.293588 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.294159 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.294312 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.299021 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.299025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.299028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.299057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.301767 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.302411 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.302416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.302419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.302569 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.302895 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.302900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.302904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.303249 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.303511 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.303738 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.303944 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:28:36.306316 29674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: Bootstrap starting.
I20260925 01:28:36.308697 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:36.309943 29674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:36.311256 29674 log.cc:824] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:36.314325 29674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: No bootstrap required, opened a new log
I20260925 01:28:36.312050 29501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f1117ff989d1419892c220fb25297f11"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54968
I20260925 01:28:36.323447 29674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.323733 29674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d12fc306e0479dbfb41ae632a1ccf4, State: Initialized, Role: FOLLOWER
I20260925 01:28:36.324175 29674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.325040 29680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.325346 29680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.326076 29674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:28:36.335145 29691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:28:36.335274 29691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:28:36.332496 29607 master_runner.cc:186] Error getting master registration for 127.6.236.190:37263: OK, instance_id { permanent_uuid: "f1117ff989d1419892c220fb25297f11" instance_seqno: 1790299715260279 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:28:36.343256 29607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:28:36.391194 29526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 4
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:28:36.407923 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.391958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38054)
0925 01:28:36.392167 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.392196 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:36.392328 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:36.392884 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:36.392889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:36.393418 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:36.393648 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.393662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.395182 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.395188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.396665 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.396680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.397667 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.397673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.397883 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.398508 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:36.398532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:36.401109 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:36.404970 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.404976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.404980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.405450 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.405835 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.405839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.405843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.405999 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.406889 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:36.406894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:36.407065 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 01:28:36.407217 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 01:28:36.407500 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:28:36.408068 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.392017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.6.236.188:45261)
0925 01:28:36.392556 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.392562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.392582 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:28:36.392640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.392646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.392718 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.392818 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.393827 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.395026 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.396808 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.397514 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.397629 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.401284 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.401292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.401300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.401340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.404848 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.405572 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.405585 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.405589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.405733 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.406101 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.406106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.406109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.406642 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.406800 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.407391 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.407633 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405}
I20260925 01:28:36.413694 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: Bootstrap starting.
I20260925 01:28:36.417538 29526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:36.418797 29526 log.cc:824] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:36.421990 29526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: No bootstrap required, opened a new log
I20260925 01:28:36.429224 29526 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.429610 29526 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1117ff989d1419892c220fb25297f11, State: Initialized, Role: FOLLOWER
I20260925 01:28:36.430086 29526 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.431269 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.431854 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.432413 29526 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:36.432523 29501 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:28:36.432708 29501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:36.436124 29501 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.438694 29501 leader_election.cc:288] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [CANDIDATE]: Term 1 election: Requested vote from peers 83b128306b064a36bfb2a96786c88ae7 (127.6.236.189:44009), 94d12fc306e0479dbfb41ae632a1ccf4 (127.6.236.188:45261)
I20260925 01:28:36.439381 29649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1117ff989d1419892c220fb25297f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94d12fc306e0479dbfb41ae632a1ccf4"
I20260925 01:28:36.439908 29649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:36.443547 29649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1117ff989d1419892c220fb25297f11 in term 1.
I20260925 01:28:36.444221 29478 leader_election.cc:302] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d12fc306e0479dbfb41ae632a1ccf4, f1117ff989d1419892c220fb25297f11; no voters:
I20260925 01:28:36.444741 29694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:36.447034 29694 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [term 1 LEADER]: Becoming Leader. State: Replica: f1117ff989d1419892c220fb25297f11, State: Running, Role: LEADER
I20260925 01:28:36.447630 29694 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.452389 29704 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1117ff989d1419892c220fb25297f11. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.452816 29704 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:36.457270 29707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:36.465487 29707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:36.476349 29600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } attempt: 5
W20260925 01:28:36.477268 29628 tablet.cc:2369] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:28:36.480440 29714 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:28:36.483569 29649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [term 1 FOLLOWER]: Refusing update from remote peer f1117ff989d1419892c220fb25297f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:36.486205 29704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 01:28:36 dist-test-slave-s0b2 krb5kdc[29446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299715, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:28:36.487682 29459 master_runner.cc:186] Error getting master registration for 127.6.236.189:44009: OK, instance_id { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" instance_seqno: 1790299715757093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:28:36.497766 29459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:28:36.500964 29480 tablet.cc:2369] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:28:36.510875 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.477004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:45261 (local address 127.0.0.1:38070)
0925 01:28:36.477397 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:36.477417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:36.477543 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:36.478443 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:36.478448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:36.479018 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:36.479243 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.479257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.480771 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.480777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.482153 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:36.482170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.483931 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.483936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:36.484210 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.484869 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:36.484914 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:36.488651 (+ 3737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:36.493016 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.493022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.493026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.493835 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.505170 (+ 11335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:36.505179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:36.505182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:36.505380 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:36.509831 (+ 4451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:36.509836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:36.510009 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 01:28:36.510160 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 01:28:36.510458 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260925 01:28:36.513422 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.477750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.6.236.188:45261)
0925 01:28:36.477920 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.477925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.477944 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:28:36.478004 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.478009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.478111 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.478209 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.479432 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.480539 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.482960 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.483648 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.483763 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.488902 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.488909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.488913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.488949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.492827 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.494025 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.494031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.494035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.499156 (+ 5121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.506108 (+ 6952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.506113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.506116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.509551 (+ 3435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.509763 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.512739 (+ 2976us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.512990 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:28:36.518548 29680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1117ff989d1419892c220fb25297f11. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.518915 29680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.519574 29600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: Bootstrap starting.
I20260925 01:28:36.524435 29600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:36.525683 29600 log.cc:824] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:36.530589 29719 mvcc.cc:205] Tried to move back new op lower bound from 7333067638704930816 to 7333067638575177728. Current Snapshot: MvccSnapshot[applied={T|T < 7333067638704930816}]
I20260925 01:28:36.533459 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.483684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.6.236.189:44009)
0925 01:28:36.483838 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.483843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.498488 (+ 14645us) server_negotiation.cc:407] Connection header received
0925 01:28:36.498746 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.498751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.498829 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.498895 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.501237 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.502421 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.504326 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.504985 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.505085 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.509018 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.509025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.509030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.509070 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.512402 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.528180 (+ 15778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.528186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.528190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.528367 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.528814 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.528820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.528824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.529184 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.529427 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.532837 (+ 3410us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.533109 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:28:36.537530 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.537874 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:36.539372 29680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.539633 29680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.543360 29707 catalog_manager.cc:1347] Generated new cluster ID: f2a7e775588b483fa21d0533b5cc4d32
I20260925 01:28:36.543469 29707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:36.545467 29600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: No bootstrap required, opened a new log
I20260925 01:28:36.550418 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.498653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.6.236.188:45261)
0925 01:28:36.514041 (+ 15388us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.514046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.520854 (+ 6808us) server_negotiation.cc:407] Connection header received
0925 01:28:36.520947 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.520956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.521052 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.521154 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.529861 (+ 8707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.530958 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.535428 (+ 4470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.536162 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.536264 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.540436 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.540443 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.540449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.540489 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.547147 (+ 6658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.547960 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.547966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.547970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.548125 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.548572 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.548576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.548579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.548931 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.549180 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.549722 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.549966 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15121,"thread_start_us":20092,"threads_started":1}
I20260925 01:28:36.556387 29600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.556779 29600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b128306b064a36bfb2a96786c88ae7, State: Initialized, Role: FOLLOWER
I20260925 01:28:36.557283 29600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } }
I20260925 01:28:36.559295 29707 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:36.560228 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.484370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.6.236.190:37263)
0925 01:28:36.484864 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.484870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.505640 (+ 20770us) server_negotiation.cc:407] Connection header received
0925 01:28:36.506444 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.506451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.506534 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.506601 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.514007 (+ 7406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.515131 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.520535 (+ 5404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.521257 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.521376 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.540679 (+ 19303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:36.540696 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:36.540701 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:36.540740 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:36.544260 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.555160 (+ 10900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.555166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.555169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.555363 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:36.556492 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:36.556497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:36.556499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:36.556877 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:36.557114 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.559599 (+ 2485us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.559882 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260925 01:28:36.563704 29707 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:36.575083 29600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:36.576278 29574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:36.570649 29724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.576635 29724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.578392 29575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1117ff989d1419892c220fb25297f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "83b128306b064a36bfb2a96786c88ae7"
I20260925 01:28:36.580483 29574 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [term 1 FOLLOWER]: Refusing update from remote peer f1117ff989d1419892c220fb25297f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:36.582093 29707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11: Generated new TSK 0
I20260925 01:28:36.583231 29707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:36.588321 29575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1117ff989d1419892c220fb25297f11 in term 1.
I20260925 01:28:36.590493 29694 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:28:36.669612 29724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1117ff989d1419892c220fb25297f11. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.669981 29724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.670301 29735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:36.670636 29735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:36.678478 29742 catalog_manager.cc:1259] Loaded cluster ID: f2a7e775588b483fa21d0533b5cc4d32
I20260925 01:28:36.678638 29742 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: loading cluster ID for follower catalog manager: success
I20260925 01:28:36.682205 29742 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: acquiring CA information for follower catalog manager: success
I20260925 01:28:36.684827 29742 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83b128306b064a36bfb2a96786c88ae7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:28:36.704807 7090 mini_kdc.cc:348] Time spent kdestroy: real 0.104s user 0.001s sys 0.022s
W20260925 01:28:36.773870 29604 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.754454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35612 (local address 127.6.236.189:44009)
0925 01:28:36.754850 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.754857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.771207 (+ 16350us) server_negotiation.cc:407] Connection header received
0925 01:28:36.772002 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.772009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.772103 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.772185 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.773504 (+ 1319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35612: BlockingRecv error: recv got EOF from 127.0.0.1:35612 (error 108)
Metrics: {"server-negotiator.queue_time_us":234}
W20260925 01:28:36.775903 29528 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.754454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.6.236.190:37263)
0925 01:28:36.754657 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.754665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.770716 (+ 16051us) server_negotiation.cc:407] Connection header received
0925 01:28:36.774143 (+ 3427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.774148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.774267 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.774351 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.775651 (+ 1300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54980: BlockingRecv error: recv got EOF from 127.0.0.1:54980 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 01:28:36.784647 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.759878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.6.236.188:45261)
0925 01:28:36.760072 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.760078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.782764 (+ 22686us) server_negotiation.cc:407] Connection header received
0925 01:28:36.783180 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.783190 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.783271 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.783336 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.784401 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38086: BlockingRecv error: recv got EOF from 127.0.0.1:38086 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:28:36.850679 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.834501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.6.236.189:44009)
0925 01:28:36.834702 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.834708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.842845 (+ 8137us) server_negotiation.cc:407] Connection header received
0925 01:28:36.843108 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.843115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.843333 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.843514 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:36.844266 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.845418 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.847532 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.848217 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.848320 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.849086 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:36.849788 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:36.849921 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.850179 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.850310 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 01:28:36.882994 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.880216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.6.236.188:45261)
0925 01:28:36.880417 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.880424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.881370 (+ 946us) server_negotiation.cc:407] Connection header received
0925 01:28:36.881617 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.881623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.881766 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.881882 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.882757 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38088: BlockingRecv error: recv got EOF from 127.0.0.1:38088 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:28:36.894119 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.887641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.6.236.189:44009)
0925 01:28:36.887856 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.887861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.888001 (+ 140us) server_negotiation.cc:407] Connection header received
0925 01:28:36.888173 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.888179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.888294 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.888398 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:36.888843 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.889987 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.891988 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.892668 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.892802 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.893312 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:36.893479 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:36.893579 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.893715 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.893805 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 01:28:36.906544 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.904635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.6.236.188:45261)
0925 01:28:36.904861 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.904867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.905008 (+ 141us) server_negotiation.cc:407] Connection header received
0925 01:28:36.905300 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.905306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.905412 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.905514 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.906331 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38096: BlockingRecv error: recv got EOF from 127.0.0.1:38096 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:28:36.918192 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.911432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.6.236.189:44009)
0925 01:28:36.911619 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.911625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.912103 (+ 478us) server_negotiation.cc:407] Connection header received
0925 01:28:36.912284 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.912290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.912388 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.912482 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:36.912950 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.914138 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.916099 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.916783 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.916904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.917369 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:36.917501 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:36.917588 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.917744 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.917854 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:28:36.927870 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.925363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.6.236.188:45261)
0925 01:28:36.925542 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.925548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.926179 (+ 631us) server_negotiation.cc:407] Connection header received
0925 01:28:36.926477 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.926483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.926600 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.926669 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.927627 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38100: BlockingRecv error: recv got EOF from 127.0.0.1:38100 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:28:36.943424 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.936740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.6.236.189:44009)
0925 01:28:36.936972 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.936978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.937508 (+ 530us) server_negotiation.cc:407] Connection header received
0925 01:28:36.937704 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.937709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.937814 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.937905 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:36.938368 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.939496 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.941377 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.942054 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.942181 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.942691 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:36.942815 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:36.942900 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.943063 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.943141 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260925 01:28:36.952088 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.950009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.6.236.188:45261)
0925 01:28:36.950320 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.950326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.950646 (+ 320us) server_negotiation.cc:407] Connection header received
0925 01:28:36.950838 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.950844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.950938 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.951032 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.951860 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38116: BlockingRecv error: recv got EOF from 127.0.0.1:38116 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 01:28:36.970698 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:36.963900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.6.236.189:44009)
0925 01:28:36.964097 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.964103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.964651 (+ 548us) server_negotiation.cc:407] Connection header received
0925 01:28:36.964835 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.964840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.964934 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.965044 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:36.965525 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.966646 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.968498 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:36.969227 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:36.969367 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:36.969836 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:36.969990 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:36.970081 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:36.970296 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 01:28:36.970379 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 01:28:36.979537 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:36.977232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.6.236.188:45261)
0925 01:28:36.977415 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:36.977421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:36.977954 (+ 533us) server_negotiation.cc:407] Connection header received
0925 01:28:36.978119 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:36.978125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:36.978259 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:36.978366 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:36.979273 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38124: BlockingRecv error: recv got EOF from 127.0.0.1:38124 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:28:37.008157 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.000561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.6.236.189:44009)
0925 01:28:37.000746 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.000752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.001265 (+ 513us) server_negotiation.cc:407] Connection header received
0925 01:28:37.001478 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.001485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.001633 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.001744 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.002314 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.003537 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.005536 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.006287 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.006419 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.006975 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.007151 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.007260 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.007499 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.007872 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 01:28:37.022749 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:37.020375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.6.236.188:45261)
0925 01:28:37.020567 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.020574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.021141 (+ 567us) server_negotiation.cc:407] Connection header received
0925 01:28:37.021363 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.021369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.021461 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.021599 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:37.022496 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38128: BlockingRecv error: recv got EOF from 127.0.0.1:38128 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:28:37.063726 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.058156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.6.236.189:44009)
0925 01:28:37.058422 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.058429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.058716 (+ 287us) server_negotiation.cc:407] Connection header received
0925 01:28:37.058937 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.058944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.059013 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.059109 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.059670 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.060586 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.061946 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.062464 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.062581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.063012 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.063116 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.063204 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.063327 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.063408 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260925 01:28:37.071412 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:37.069578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.6.236.188:45261)
0925 01:28:37.069785 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.069791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.070111 (+ 320us) server_negotiation.cc:407] Connection header received
0925 01:28:37.070279 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.070285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.070385 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.070501 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:37.071178 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38138: BlockingRecv error: recv got EOF from 127.0.0.1:38138 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:28:37.143282 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.138343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.6.236.189:44009)
0925 01:28:37.138543 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.138550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.138892 (+ 342us) server_negotiation.cc:407] Connection header received
0925 01:28:37.139096 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.139100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.139182 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.139256 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.139669 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.140360 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.141666 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.142065 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.142181 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.142625 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.142729 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.142803 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.142937 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.143017 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 01:28:37.149855 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:37.148121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.6.236.188:45261)
0925 01:28:37.148322 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.148328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.148566 (+ 238us) server_negotiation.cc:407] Connection header received
0925 01:28:37.148739 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.148744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.148818 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.148900 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:37.149642 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:28:37.177294 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1117ff989d1419892c220fb25297f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1117ff989d1419892c220fb25297f11" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 37263 } } peers { permanent_uuid: "83b128306b064a36bfb2a96786c88ae7" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 44009 } } peers { permanent_uuid: "94d12fc306e0479dbfb41ae632a1ccf4" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 45261 } } }
I20260925 01:28:37.177605 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1117ff989d1419892c220fb25297f11 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:37.285816 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.280851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.6.236.189:44009)
0925 01:28:37.281028 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.281035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.281373 (+ 338us) server_negotiation.cc:407] Connection header received
0925 01:28:37.281581 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.281585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.281654 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.281734 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.282146 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.282859 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.284218 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.284624 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.284733 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.285148 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.285272 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.285348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.285489 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.285570 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 01:28:37.292675 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:37.291064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.6.236.188:45261)
0925 01:28:37.291223 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.291227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.291584 (+ 357us) server_negotiation.cc:407] Connection header received
0925 01:28:37.291734 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.291738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.291805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.291884 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:37.292502 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38160: BlockingRecv error: recv got EOF from 127.0.0.1:38160 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:28:37.337675 29691 catalog_manager.cc:1259] Loaded cluster ID: f2a7e775588b483fa21d0533b5cc4d32
I20260925 01:28:37.337790 29691 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: loading cluster ID for follower catalog manager: success
I20260925 01:28:37.340350 29691 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: acquiring CA information for follower catalog manager: success
I20260925 01:28:37.342079 29691 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94d12fc306e0479dbfb41ae632a1ccf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:28:37.558002 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.553159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.6.236.189:44009)
0925 01:28:37.553403 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.553410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.553587 (+ 177us) server_negotiation.cc:407] Connection header received
0925 01:28:37.553794 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.553799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.553882 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.553961 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.554423 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.555139 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.556459 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.556855 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.556965 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.557346 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.557450 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.557526 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.557672 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.557752 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 01:28:37.568794 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.563561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.6.236.188:45261)
0925 01:28:37.563754 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.563760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.563903 (+ 143us) server_negotiation.cc:407] Connection header received
0925 01:28:37.564088 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.564093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.564269 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.564367 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.564740 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.565428 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.566740 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.567116 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.567235 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.567692 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.568220 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.568365 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.568467 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.568549 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:28:37.591826 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.574636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.6.236.188:45261)
0925 01:28:37.574846 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.574852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.585914 (+ 11062us) server_negotiation.cc:407] Connection header received
0925 01:28:37.586714 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.586718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.586797 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.586885 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.587396 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.588207 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.589818 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.590413 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.590534 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.591088 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.591219 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.591268 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.591468 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.591550 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:28:37.591912 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.574732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.6.236.189:44009)
0925 01:28:37.574910 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.574916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.585875 (+ 10959us) server_negotiation.cc:407] Connection header received
0925 01:28:37.586041 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.586047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.586135 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.586233 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.586813 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.587690 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.589950 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.590586 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.590700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.591177 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.591301 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.591371 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.591560 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.591640 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:28:37.593267 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:37.574967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.6.236.190:37263)
0925 01:28:37.585384 (+ 10417us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:37.585390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:37.585411 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:28:37.585532 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:37.585537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:37.585771 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:37.585951 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:37.586575 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.587595 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.590346 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:37.590923 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:37.591037 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:37.591543 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:37.592563 (+ 1020us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:37.592685 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:37.592778 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 01:28:37.592863 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10290,"thread_start_us":10188,"threads_started":1}
I20260925 01:28:37.597563 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29459
I20260925 01:28:37.598567 29521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:37.706494 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29459
W20260925 01:28:37.724000 29625 connection.cc:535] server connection from 127.0.0.1:38054 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:28:37.724392 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29533
I20260925 01:28:37.725423 29595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:37.830430 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29533
I20260925 01:28:37.847659 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29607
I20260925 01:28:37.848757 29669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:37.954733 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29607
2026-09-25T01:28:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4383 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:38 dist-test-slave-s0b2 krb5kdc[29867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:38 dist-test-slave-s0b2 krb5kdc[29867](info): set up 2 sockets
Sep 25 01:28:38 dist-test-slave-s0b2 krb5kdc[29867](info): commencing operation
krb5kdc: starting...
W20260925 01:28:38.987371 7090 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/test-user.keytab.
Sep 25 01:28:39 dist-test-slave-s0b2 krb5kdc[29867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.236.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
Entry for principal kudu/127.6.236.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab.
I20260925 01:28:39.235559 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33291
--webserver_interface=127.6.236.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:39.565322 29880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:39.565653 29880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:39.565724 29880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:39.565804 29880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:39.573949 29880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:39.574049 29880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:39.574106 29880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:39.574156 29880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:39.574219 29880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:39.584913 29880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.190.keytab
--principal=kudu/127.6.236.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.190:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.236.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:39.586182 29880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:39.587524 29880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:39.599212 29885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:39.599170 29886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:39.599133 29888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:39.600302 29880 server_base.cc:1034] running on GCE node
I20260925 01:28:39.601091 29880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:39.602528 29880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:39.603763 29880 hybrid_clock.cc:648] HybridClock initialized: now 1790299719603726 us; error 73 us; skew 500 ppm
I20260925 01:28:39.604071 29880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:39 dist-test-slave-s0b2 krb5kdc[29867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:39.608017 29880 init.cc:377] Logged in from keytab as kudu/127.6.236.190@KRBTEST.COM (short username kudu)
I20260925 01:28:39.609464 29880 webserver.cc:460] Webserver started at http://127.6.236.190:33515/ using document root <none> and password file <none>
I20260925 01:28:39.610131 29880 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:39.610291 29880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:39.610591 29880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:39.613747 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data/instance:
uuid: "b60c593e2c9d4411b52d659d3925bfcd"
format_stamp: "Formatted at 2026-09-25 01:28:39 on dist-test-slave-s0b2"
server_key: "d7cec8f99c58519ff6f269bed13b1d0e"
server_key_iv: "1ff435cbe272ad44c4e352c19aceeb2c"
server_key_version: "encryptionkey@0"
I20260925 01:28:39.614522 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance:
uuid: "b60c593e2c9d4411b52d659d3925bfcd"
format_stamp: "Formatted at 2026-09-25 01:28:39 on dist-test-slave-s0b2"
server_key: "d7cec8f99c58519ff6f269bed13b1d0e"
server_key_iv: "1ff435cbe272ad44c4e352c19aceeb2c"
server_key_version: "encryptionkey@0"
I20260925 01:28:39.619740 29880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:28:39.623282 29895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:39.624732 29880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:28:39.624897 29880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal
uuid: "b60c593e2c9d4411b52d659d3925bfcd"
format_stamp: "Formatted at 2026-09-25 01:28:39 on dist-test-slave-s0b2"
server_key: "d7cec8f99c58519ff6f269bed13b1d0e"
server_key_iv: "1ff435cbe272ad44c4e352c19aceeb2c"
server_key_version: "encryptionkey@0"
I20260925 01:28:39.625106 29880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:39.644063 29880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:39.648279 29880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:39.648524 29880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:39.665647 29880 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.190:33291
I20260925 01:28:39.665655 29946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.190:33291 every 8 connection(s)
I20260925 01:28:39.667057 29880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:39.669454 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29880
I20260925 01:28:39.669615 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-0/wal/instance
I20260925 01:28:39.670058 7090 external_mini_cluster.cc:1428] Setting key fde4e2d3b6727bb5dcd84394fb113724
I20260925 01:28:39.670961 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:39.677645 29947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:39 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.189@KRBTEST.COM" created.
I20260925 01:28:39.700389 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.681578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33291 (local address 127.0.0.1:36064)
0925 01:28:39.682433 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.682478 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:39.684444 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:39.685282 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:39.685289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:39.685969 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:39.686358 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:39.686378 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:39.687953 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:39.687957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:39.689019 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:39.689031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:39.690044 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:39.690051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:39.690324 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:39.690980 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:39.691003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:39.693304 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:39.696456 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:39.696467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:39.696477 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:39.696886 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:39.697351 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:39.697355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:39.697357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:39.697511 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:39.698450 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:39.698456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:39.698670 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 01:28:39.698878 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 01:28:39.699305 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":35,"thread_start_us":324,"threads_started":1}
I20260925 01:28:39.700433 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.681579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.6.236.190:33291)
0925 01:28:39.683227 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:39.683238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:39.683287 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:28:39.684809 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:39.684814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:39.685058 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:39.685244 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:39.686563 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:39.687804 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:39.689191 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:39.689813 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:39.689982 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:39.693513 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:39.693520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:39.693535 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:39.693564 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:39.696318 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:39.697038 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:39.697048 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:39.697051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:39.697241 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:39.697609 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:39.697614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:39.697617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:39.698322 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:39.698515 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:39.699056 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 01:28:39.699447 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":1427,"thread_start_us":1217,"threads_started":1}
I20260925 01:28:39.703356 29947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:39.704416 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.704036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53814)
0925 01:28:39.704205 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.704312 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 01:28:39.704664 29898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
I20260925 01:28:39.706106 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.705786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53818)
0925 01:28:39.705922 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.705983 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 01:28:39.706350 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:41105: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
Entry for principal kudu/127.6.236.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab.
I20260925 01:28:39.726526 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:41105
--webserver_interface=127.6.236.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:28:39.755681 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } attempt: 1
I20260925 01:28:39.756904 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.756386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53826)
0925 01:28:39.756599 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.756746 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:28:39.758020 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.757512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53842)
0925 01:28:39.757645 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.757793 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 01:28:39.758203 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:41105: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
I20260925 01:28:39.826503 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } attempt: 2
I20260925 01:28:39.827812 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.827287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53846)
0925 01:28:39.827507 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.827630 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:28:39.829094 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.828567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53852)
0925 01:28:39.828759 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.828907 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:28:39.829358 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:41105: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
I20260925 01:28:39.920631 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } attempt: 3
I20260925 01:28:39.921934 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.921481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53862)
0925 01:28:39.921707 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.921800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:28:39.923065 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:39.922710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53876)
0925 01:28:39.922849 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:39.922921 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 01:28:39.923267 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:41105: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
W20260925 01:28:40.048399 29954 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:40.048696 29954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:40.048766 29954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:40.048844 29954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:40.055789 29954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:40.055868 29954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:40.055922 29954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:40.055970 29954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:40.056030 29954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:40.065431 29954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.189.keytab
--principal=kudu/127.6.236.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.189:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.236.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:40.066474 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } attempt: 4
I20260925 01:28:40.066627 29954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:40.067791 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.067234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53888)
0925 01:28:40.067530 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.067630 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260925 01:28:40.068215 29954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:28:40.068848 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.068516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53904)
0925 01:28:40.068639 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.068735 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 01:28:40.069021 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.189:41105: Network error: Client connection negotiation failed: client connection to 127.6.236.189:41105: connect: Connection refused (error 111)
W20260925 01:28:40.079434 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:40.079466 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:40.079496 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:40.080500 29954 server_base.cc:1034] running on GCE node
I20260925 01:28:40.081190 29954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:40.082540 29954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:40.083767 29954 hybrid_clock.cc:648] HybridClock initialized: now 1790299720083698 us; error 40 us; skew 500 ppm
I20260925 01:28:40.084064 29954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:40.086975 29954 init.cc:377] Logged in from keytab as kudu/127.6.236.189@KRBTEST.COM (short username kudu)
I20260925 01:28:40.088383 29954 webserver.cc:460] Webserver started at http://127.6.236.189:36473/ using document root <none> and password file <none>
I20260925 01:28:40.089025 29954 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:40.089126 29954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:40.089421 29954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:40.092275 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data/instance:
uuid: "e1e388f3e13b40b585ebf0357527af4b"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "c58aa2ee3f6ce94b7097dd7b9ce58c46"
server_key_iv: "b263443c614c29b5cd6d78c7ba89e4c1"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.092999 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal/instance:
uuid: "e1e388f3e13b40b585ebf0357527af4b"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "c58aa2ee3f6ce94b7097dd7b9ce58c46"
server_key_iv: "b263443c614c29b5cd6d78c7ba89e4c1"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.097741 29954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 01:28:40.100903 29969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:40.102151 29954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:28:40.102344 29954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal
uuid: "e1e388f3e13b40b585ebf0357527af4b"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "c58aa2ee3f6ce94b7097dd7b9ce58c46"
server_key_iv: "b263443c614c29b5cd6d78c7ba89e4c1"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.102530 29954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:40.124598 29954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:40.128456 29954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:40.128695 29954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:40.145285 29954 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.189:41105
I20260925 01:28:40.145308 30020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.189:41105 every 8 connection(s)
I20260925 01:28:40.146610 29954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/data/info.pb
I20260925 01:28:40.149130 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 29954
I20260925 01:28:40.149314 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-1/wal/instance
I20260925 01:28:40.149852 7090 external_mini_cluster.cc:1428] Setting key efa088c41546c3615abdf751b6cfa66c
I20260925 01:28:40.150627 30021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:40.158720 30021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.236.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.236.188@KRBTEST.COM" created.
I20260925 01:28:40.178786 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.161853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.6.236.190:33291)
0925 01:28:40.162025 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.162030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.163332 (+ 1302us) server_negotiation.cc:407] Connection header received
0925 01:28:40.165153 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.165159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.165239 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.165335 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.166858 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.167625 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.169109 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.169552 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.169664 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.173160 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.173165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.173168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.173201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.175823 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.176523 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.176527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.176529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.176648 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.177050 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.177053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.177055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.177356 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.177489 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.178263 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.178500 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:28:40.180476 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.162477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33291 (local address 127.0.0.1:36074)
0925 01:28:40.163177 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.163215 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.164886 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.165566 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.165574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.166092 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.166686 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.166703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.167786 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.167790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.168953 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.168973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.169779 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.169787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.170005 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.170643 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.170666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.172935 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.175950 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.175959 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.175976 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.176401 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.176753 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.176760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.176764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.176936 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.177476 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.177484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.177835 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.179055 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.179466 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":270,"threads_started":1}
I20260925 01:28:40.182228 30021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:40.196466 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.183123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53908)
0925 01:28:40.183305 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.183322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.183422 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.184352 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.184360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.184761 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.184924 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.184934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.186451 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.186455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.187304 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.187315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.188334 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.188338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.188466 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.188955 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.188971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.191009 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.194429 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.194433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.194435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.194771 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.195119 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.195122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.195124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.195245 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.195843 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.195847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.195955 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.196052 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.196225 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:28:40.196797 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.183350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.6.236.189:41105)
0925 01:28:40.183950 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.183956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.183981 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:28:40.184022 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.184028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.184200 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.184313 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.185076 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.186327 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.187431 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.188209 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.188327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.191217 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.191231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.191247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.191280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.194323 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.194888 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.194896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.194899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.195035 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.195338 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.195342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.195346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.195719 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.195919 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.196203 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.196428 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":317,"threads_started":1}
I20260925 01:28:40.199227 30021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:40.200533 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.200001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46382)
0925 01:28:40.200232 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.200373 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 01:28:40.200734 29971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.202296 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.201941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46386)
0925 01:28:40.202093 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.202161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:28:40.202507 30021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
Entry for principal kudu/127.6.236.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab.
I20260925 01:28:40.211357 7090 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgybJNj/build/asan/bin/kudu
/tmp/dist-test-taskgybJNj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:42307
--webserver_interface=127.6.236.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.236.148:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:28:40.251793 30021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 1
I20260925 01:28:40.253199 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.252663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46392)
0925 01:28:40.252922 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.253030 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 01:28:40.254348 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.253974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46394)
0925 01:28:40.254120 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.254194 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 01:28:40.254545 30021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.322785 30021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 2
I20260925 01:28:40.324011 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.323555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46400)
0925 01:28:40.323765 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.323858 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:28:40.324993 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.324694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46416)
0925 01:28:40.324810 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.324879 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 01:28:40.325201 30021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.368248 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } attempt: 5
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:40.386768 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.369105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53914)
0925 01:28:40.369343 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.369366 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.369489 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.369884 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.369887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.370299 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.370487 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.370496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.371886 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.371890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.372734 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.372745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.373674 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.373678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.373835 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.377947 (+ 4112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.377973 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.380705 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.384432 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.384436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.384439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.384833 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.385238 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.385242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.385245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.385385 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.385988 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.385992 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.386129 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.386283 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.386514 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 01:28:40.386953 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.369300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.6.236.189:41105)
0925 01:28:40.369506 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.369534 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.369558 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:28:40.369626 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.369631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.369738 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.369845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.370634 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.371752 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.372881 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.373549 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.373676 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.380946 (+ 7270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.380954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.380959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.381008 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.384272 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.384956 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.384962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.384965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.385118 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.385486 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.385491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.385494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.385887 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.386036 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.386388 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.386630 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:28:40.388025 29947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:28:40.388942 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.388575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46424)
0925 01:28:40.388737 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.388809 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 01:28:40.389803 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.389500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46430)
0925 01:28:40.389630 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.389710 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 01:28:40.390009 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.416392 30021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 3
I20260925 01:28:40.417640 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.417196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46434)
0925 01:28:40.417389 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.417481 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 01:28:40.418649 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.418213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46448)
0925 01:28:40.418456 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.418532 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260925 01:28:40.418845 30021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.441215 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 1
I20260925 01:28:40.442364 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.441888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46452)
0925 01:28:40.442082 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.442168 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:28:40.443318 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.442994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46456)
0925 01:28:40.443140 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.443215 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 01:28:40.443502 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.511716 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 2
I20260925 01:28:40.512778 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.512318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46460)
0925 01:28:40.512519 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.512603 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:28:40.513782 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.513457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46472)
0925 01:28:40.513600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.513664 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 01:28:40.513972 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
W20260925 01:28:40.531792 30028 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:28:40.532112 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:40.532207 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:40.532327 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:40.539355 30028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:40.539444 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:40.539531 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:40.539593 30028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:40.539677 30028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:40.549520 30028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.236.148:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.236.190:33291,127.6.236.189:41105,127.6.236.188:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/krb5kdc/kudu_127.6.236.188.keytab
--principal=kudu/127.6.236.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.236.188:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.236.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:40.550797 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:40.552075 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:28:40.562026 30021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 4
W20260925 01:28:40.563050 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:40.563093 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:40.563488 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.562807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46484)
0925 01:28:40.563192 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.563299 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
W20260925 01:28:40.563526 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:40.564401 30028 server_base.cc:1034] running on GCE node
I20260925 01:28:40.564548 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.564189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46498)
0925 01:28:40.564342 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.564423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 01:28:40.564822 30021 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.565244 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:40.566586 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:40.567844 30028 hybrid_clock.cc:648] HybridClock initialized: now 1790299720567773 us; error 39 us; skew 500 ppm
I20260925 01:28:40.568166 30028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:40.571074 30028 init.cc:377] Logged in from keytab as kudu/127.6.236.188@KRBTEST.COM (short username kudu)
I20260925 01:28:40.572453 30028 webserver.cc:460] Webserver started at http://127.6.236.188:35053/ using document root <none> and password file <none>
I20260925 01:28:40.573128 30028 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:40.573211 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:40.573474 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:40.576349 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data/instance:
uuid: "c03ec591990a400387a1aad21b889767"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "32261215247890179bdb95812d26e677"
server_key_iv: "54b291b84d235ee61b7230b21a636180"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.577068 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal/instance:
uuid: "c03ec591990a400387a1aad21b889767"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "32261215247890179bdb95812d26e677"
server_key_iv: "54b291b84d235ee61b7230b21a636180"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.582023 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:28:40.585335 30043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:40.586680 30028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:40.586830 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data,/tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal
uuid: "c03ec591990a400387a1aad21b889767"
format_stamp: "Formatted at 2026-09-25 01:28:40 on dist-test-slave-s0b2"
server_key: "32261215247890179bdb95812d26e677"
server_key_iv: "54b291b84d235ee61b7230b21a636180"
server_key_version: "encryptionkey@0"
I20260925 01:28:40.587005 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:40.620164 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 3
I20260925 01:28:40.621308 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.620828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46506)
0925 01:28:40.621049 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.621141 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:28:40.622455 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.622041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46514)
0925 01:28:40.622273 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.622351 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 01:28:40.622651 29947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.236.188:42307: Network error: Client connection negotiation failed: client connection to 127.6.236.188:42307: connect: Connection refused (error 111)
I20260925 01:28:40.648162 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:40.652143 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:40.652395 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:40.669742 30028 rpc_server.cc:307] RPC server started. Bound to: 127.6.236.188:42307
I20260925 01:28:40.669763 30094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.236.188:42307 every 8 connection(s)
I20260925 01:28:40.671062 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/data/info.pb
I20260925 01:28:40.674285 7090 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgybJNj/build/asan/bin/kudu as pid 30028
I20260925 01:28:40.674465 7090 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgybJNj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299375376939-7090-0/minicluster-data/master-2/wal/instance
I20260925 01:28:40.674750 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:40.675007 7090 external_mini_cluster.cc:1428] Setting key 180c383f0e52ba3db1f1bfab070ccc5d
I20260925 01:28:40.683768 30095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
I20260925 01:28:40.697625 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.676721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.6.236.188:42307)
0925 01:28:40.677453 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.677464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.677507 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:28:40.679265 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.679296 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.679699 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.680032 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.681693 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.682990 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.684719 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.685399 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.685526 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.689941 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.689947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.689957 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.689990 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.693350 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.693981 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.693990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.693998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.694195 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.694591 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.694596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.694599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.695080 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.695332 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.695944 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.696343 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":313,"threads_started":1}
I20260925 01:28:40.702080 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.686574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.6.236.190:33291)
0925 01:28:40.687920 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.687924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.687940 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:28:40.687989 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.687992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.688050 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.688138 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.689306 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.690260 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.692182 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.692610 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.692729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.696105 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.696112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.696116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.696155 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.699463 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.700207 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.700212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.700216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.700367 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.700713 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.700718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.700722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.701082 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.701243 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.701657 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.701810 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":1085,"threads_started":1}
I20260925 01:28:40.702296 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.686534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.190:33291 (local address 127.0.0.1:36088)
0925 01:28:40.687210 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.687255 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.687397 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.688245 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.688253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.688847 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.689136 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.689147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.690408 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.690413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.692029 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.692044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.692795 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.692802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.692988 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.693565 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.693589 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.695899 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.699611 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.699620 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.699626 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.700095 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.700467 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.700472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.700476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.700611 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.701209 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.701216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.701361 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.701566 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.701891 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260925 01:28:40.703642 30095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.190@KRBTEST.COM
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:40.715121 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.700523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.6.236.190:33291)
0925 01:28:40.701141 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.701147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.701179 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:28:40.701224 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.701228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.701294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.701375 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.702851 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.703909 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.705489 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.705973 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.706088 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.709966 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.709971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.709974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.710002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.712704 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.713317 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.713322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.713324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.713443 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.713780 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.713784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.713787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.714035 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.714294 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.714592 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.714804 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":313,"threads_started":1}
I20260925 01:28:40.715741 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.704534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.189:41105 (local address 127.0.0.1:53922)
0925 01:28:40.704795 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.704812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.704911 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.705186 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.705191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.705599 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.705736 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.705744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.706824 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.706828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.707686 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.707698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.708347 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.708352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.708529 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.708988 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.709009 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.710952 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.713571 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.713576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.713580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.713908 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.714274 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.714279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.714283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.714413 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.714879 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.714884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.715034 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.715165 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.715421 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 01:28:40.715808 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.704520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.6.236.189:41105)
0925 01:28:40.704664 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.704669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.704870 (+ 201us) server_negotiation.cc:407] Connection header received
0925 01:28:40.705014 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.705018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.705078 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.705148 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.705880 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.706654 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.707817 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.708211 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.708317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.711125 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.711129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.711132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.711163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.713452 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.714014 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.714020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.714023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.714150 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.714512 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.714517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.714520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.714761 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.715061 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.715258 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.715455 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:28:40.716791 30095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.188@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.236.189@KRBTEST.COM
I20260925 01:28:40.728597 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.717366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46530)
0925 01:28:40.717487 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.717503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.717576 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.717832 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.717835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.718120 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.718297 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.718309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.719273 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.719278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.720078 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.720089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.720720 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.720725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.720892 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.721339 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.721354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.723246 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.726295 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.726301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.726304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.726667 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.727046 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.727050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.727052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.727167 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.727724 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.727728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.727870 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.728019 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.728307 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 01:28:40.728871 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.717432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.6.236.188:42307)
0925 01:28:40.717574 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.717579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.717597 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:28:40.717643 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.717647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.717713 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.717826 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.718429 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.719154 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.720198 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.720581 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.720682 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.723398 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.723403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.723406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.723433 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.726124 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.726785 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.726791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.726794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.726936 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.727278 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.727283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.727286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.727617 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.727773 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.728113 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.728494 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":50}
I20260925 01:28:40.730417 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.716680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.6.236.189:41105)
0925 01:28:40.716828 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.716833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.717080 (+ 247us) server_negotiation.cc:407] Connection header received
0925 01:28:40.717123 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.717128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.717196 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.717293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.718657 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.719324 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.720500 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.720930 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.721032 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.724796 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.724802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.724808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.724845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.727664 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.728367 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.728372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.728375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.728528 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.729070 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.729076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.729079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.729383 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.729579 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.729911 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.730118 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:28:40.734434 30095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: Bootstrap starting.
I20260925 01:28:40.734844 7090 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:28:40.738070 30095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:40.739096 30095 log.cc:824] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:40.737705 29922 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b60c593e2c9d4411b52d659d3925bfcd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36094
I20260925 01:28:40.741796 30095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: No bootstrap required, opened a new log
I20260925 01:28:40.748556 30095 raft_consensus.cc:348] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.748848 30095 raft_consensus.cc:729] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03ec591990a400387a1aad21b889767, State: Initialized, Role: FOLLOWER
I20260925 01:28:40.749264 30095 consensus_queue.cc:260] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.750103 30103 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:40.750509 30103 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:40.751008 30095 sys_catalog.cc:564] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:28:40.759640 30114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:28:40.759759 30114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:28:40.757563 30028 master_runner.cc:186] Error getting master registration for 127.6.236.190:33291: OK, instance_id { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" instance_seqno: 1790299719660792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:28:40.768074 30028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:28:40.799834 29947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 4
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299719, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.190@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:28:40.815740 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.800684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46542)
0925 01:28:40.800898 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.800920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.801045 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.801399 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.801404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.801915 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.802121 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.802132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.803590 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.803595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.805061 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.805076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.806083 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.806089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.806298 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.806936 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.806953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.809101 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.813086 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.813090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.813092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.813564 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.813946 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.813951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.813954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.814114 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.814761 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.814766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.814949 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.815089 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.815355 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:28:40.815781 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.800811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.6.236.188:42307)
0925 01:28:40.801026 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.801032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.801058 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:28:40.801161 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.801168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.801250 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.801354 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.802302 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.803451 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.805212 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.805934 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.806061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.809321 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.809328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.809334 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.809378 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.812967 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.813678 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.813684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.813687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.813843 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.814217 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.814223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.814226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.814645 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.814838 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.815206 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.815415 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:28:40.820420 29947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: Bootstrap starting.
I20260925 01:28:40.824373 29947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:40.825615 29947 log.cc:824] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:40.828316 29947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: No bootstrap required, opened a new log
I20260925 01:28:40.835637 29947 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.836057 29947 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60c593e2c9d4411b52d659d3925bfcd, State: Initialized, Role: FOLLOWER
I20260925 01:28:40.836485 29947 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.837318 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:40.837702 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:40.838510 29947 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:40.838680 29922 raft_consensus.cc:482] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:28:40.838922 29922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:40.842290 29922 raft_consensus.cc:504] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.844751 30070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b60c593e2c9d4411b52d659d3925bfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c03ec591990a400387a1aad21b889767"
I20260925 01:28:40.845243 30070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:40.845309 29922 leader_election.cc:288] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [CANDIDATE]: Term 1 election: Requested vote from peers e1e388f3e13b40b585ebf0357527af4b (127.6.236.189:41105), c03ec591990a400387a1aad21b889767 (127.6.236.188:42307)
I20260925 01:28:40.848908 30070 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60c593e2c9d4411b52d659d3925bfcd in term 1.
I20260925 01:28:40.849507 29897 leader_election.cc:302] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b60c593e2c9d4411b52d659d3925bfcd, c03ec591990a400387a1aad21b889767; no voters:
I20260925 01:28:40.849958 30117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:40.861323 30117 raft_consensus.cc:686] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [term 1 LEADER]: Becoming Leader. State: Replica: b60c593e2c9d4411b52d659d3925bfcd, State: Running, Role: LEADER
I20260925 01:28:40.861984 30117 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.864029 30021 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } attempt: 5
I20260925 01:28:40.869423 30124 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60c593e2c9d4411b52d659d3925bfcd. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:40.869805 30124 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:40.873811 30128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 01:28:40 dist-test-slave-s0b2 krb5kdc[29867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299720, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.236.189@KRBTEST.COM for kudu/127.6.236.188@KRBTEST.COM
I20260925 01:28:40.882591 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.864830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.236.188:42307 (local address 127.0.0.1:46552)
0925 01:28:40.865042 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 01:28:40.865066 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:28:40.865209 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:28:40.865876 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:28:40.865881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:28:40.866403 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:28:40.866639 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.866652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.868196 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.868202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.869894 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:28:40.869912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.871189 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.871195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:28:40.871417 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.872110 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:28:40.872136 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:28:40.875021 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:28:40.879388 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.879397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.879402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.879965 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.880485 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:28:40.880491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:28:40.880495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:28:40.880748 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:28:40.881484 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:28:40.881489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:28:40.881677 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 01:28:40.881833 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 01:28:40.882147 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:28:40.883572 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.865364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.6.236.188:42307)
0925 01:28:40.865542 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.865549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.865571 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:28:40.865639 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.865645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.865732 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.865849 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.866851 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.868015 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.870080 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.870844 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.870965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.875250 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.875257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.875263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.875303 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.879118 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.880150 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.880157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.880161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.880340 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.880891 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.880896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.880900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.881282 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.881421 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.882956 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.883200 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:28:40.889690 30021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: Bootstrap starting.
I20260925 01:28:40.894641 30021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:40.896070 30021 log.cc:824] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: Log is configured to *not* fsync() on all Append() calls
W20260925 01:28:40.901579 29901 tablet.cc:2369] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:28:40.904474 30049 tablet.cc:2369] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:28:40.925755 30128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:40.937273 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.878579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.6.236.190:33291)
0925 01:28:40.878770 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.878776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.900794 (+ 22018us) server_negotiation.cc:407] Connection header received
0925 01:28:40.901148 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.901155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.901244 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.901323 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.913256 (+ 11933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.914412 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.916239 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.916913 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.917011 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.923314 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.923322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.923327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.923378 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.927246 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.930638 (+ 3392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.930644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.930648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.930832 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.934890 (+ 4058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.934896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.934900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.935311 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.935600 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.936608 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.936880 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:28:40.948006 30021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: No bootstrap required, opened a new log
I20260925 01:28:40.955111 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.879356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.6.236.189:41105)
0925 01:28:40.879533 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.879548 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.904969 (+ 25421us) server_negotiation.cc:407] Connection header received
0925 01:28:40.905230 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.905235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.905316 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.905384 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.907307 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.908459 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.927985 (+ 19526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.928733 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.928841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.941738 (+ 12897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.941745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.941751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.941795 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.945213 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.951795 (+ 6582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.951801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.951806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.951994 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.952542 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.952548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.952551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.952922 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.953173 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.953803 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.954043 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:28:40.963127 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:40.910856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.6.236.188:42307)
0925 01:28:40.911094 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:40.911101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:40.926791 (+ 15690us) server_negotiation.cc:407] Connection header received
0925 01:28:40.927049 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:40.927080 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:40.927182 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:40.927268 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:40.931578 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.932711 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.939312 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:40.939997 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:40.940123 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:40.954197 (+ 14074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:40.954205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:40.954211 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:40.954289 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:40.958465 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.959304 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.959311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.959314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.959484 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:40.959950 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:40.959955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:40.959958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:40.960320 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:40.960634 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:40.962449 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0925 01:28:40.962717 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:28:40.971378 30021 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.971858 30021 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e388f3e13b40b585ebf0357527af4b, State: Initialized, Role: FOLLOWER
I20260925 01:28:40.972424 30021 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } }
I20260925 01:28:40.978816 30070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [term 1 FOLLOWER]: Refusing update from remote peer b60c593e2c9d4411b52d659d3925bfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:40.979882 30124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:28:41.006671 30103 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60c593e2c9d4411b52d659d3925bfcd. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.007009 30103 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:41.016196 30137 mvcc.cc:205] Tried to move back new op lower bound from 7333067657099096064 to 7333067656656642048. Current Snapshot: MvccSnapshot[applied={T|T < 7333067657099096064}]
I20260925 01:28:41.021945 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.022437 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:41.023691 30140 mvcc.cc:205] Tried to move back new op lower bound from 7333067657099096064 to 7333067656656642048. Current Snapshot: MvccSnapshot[applied={T|T < 7333067657099096064}]
I20260925 01:28:41.024559 30128 catalog_manager.cc:1347] Generated new cluster ID: 45838dcf266a4e128fdec8db46ed5744
I20260925 01:28:41.024642 30128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:41.029686 30103 sys_catalog.cc:455] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.029937 30103 sys_catalog.cc:458] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:41.051440 30139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.052578 30139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:41.053121 30021 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:41.053514 29995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:28:41.057216 29995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [term 1 FOLLOWER]: Refusing update from remote peer b60c593e2c9d4411b52d659d3925bfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:28:41.057857 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b60c593e2c9d4411b52d659d3925bfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1e388f3e13b40b585ebf0357527af4b"
I20260925 01:28:41.060243 30124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:28:41.061239 29996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60c593e2c9d4411b52d659d3925bfcd in term 1.
I20260925 01:28:41.080852 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.081192 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:41.082100 30148 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:28:41.083120 30139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60c593e2c9d4411b52d659d3925bfcd. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.083534 30139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:41.083879 30128 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:41.085641 30128 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:41.085961 29880 master_runner.cc:186] Error getting master registration for 127.6.236.189:41105: OK, instance_id { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" instance_seqno: 1790299720140596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:28:41.099359 29880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:28:41.100960 30153 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60c593e2c9d4411b52d659d3925bfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b60c593e2c9d4411b52d659d3925bfcd" member_type: VOTER last_known_addr { host: "127.6.236.190" port: 33291 } } peers { permanent_uuid: "e1e388f3e13b40b585ebf0357527af4b" member_type: VOTER last_known_addr { host: "127.6.236.189" port: 41105 } } peers { permanent_uuid: "c03ec591990a400387a1aad21b889767" member_type: VOTER last_known_addr { host: "127.6.236.188" port: 42307 } } }
I20260925 01:28:41.102211 30153 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:28:41.109781 30128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b60c593e2c9d4411b52d659d3925bfcd: Generated new TSK 0
I20260925 01:28:41.110934 30128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:28:41.129532 30162 catalog_manager.cc:1259] Loaded cluster ID: 45838dcf266a4e128fdec8db46ed5744
I20260925 01:28:41.129798 30162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: loading cluster ID for follower catalog manager: success
I20260925 01:28:41.133478 30162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: acquiring CA information for follower catalog manager: success
I20260925 01:28:41.136001 30162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1e388f3e13b40b585ebf0357527af4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:28:41.268033 30101 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.244859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.6.236.190:33291)
0925 01:28:41.245048 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.245069 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.265856 (+ 20787us) server_negotiation.cc:407] Connection header received
0925 01:28:41.266429 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.266435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.266526 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.266605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.267749 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36116: BlockingRecv error: recv got EOF from 127.0.0.1:36116 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:28:41.270639 30025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.251105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.6.236.189:41105)
0925 01:28:41.251326 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.251333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.268739 (+ 17406us) server_negotiation.cc:407] Connection header received
0925 01:28:41.269011 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.269017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.269132 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.269203 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.270417 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53948: BlockingRecv error: recv got EOF from 127.0.0.1:53948 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:28:41.273490 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.271821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.6.236.188:42307)
0925 01:28:41.272000 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.272006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.272189 (+ 183us) server_negotiation.cc:407] Connection header received
0925 01:28:41.272382 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.272387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.272456 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.272516 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.273272 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46566: BlockingRecv error: recv got EOF from 127.0.0.1:46566 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:28:41.292623 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.286664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.6.236.189:41105)
0925 01:28:41.286854 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.286861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.287289 (+ 428us) server_negotiation.cc:407] Connection header received
0925 01:28:41.287535 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.287539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.287698 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.287835 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.288238 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.288987 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.290374 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.290818 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.290935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.291404 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.291971 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.292108 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.292223 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.292306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:28:41.299829 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.298279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.6.236.188:42307)
0925 01:28:41.298440 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.298446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.298464 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:28:41.298663 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.298667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.298769 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.298830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.299665 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46576: BlockingRecv error: recv got EOF from 127.0.0.1:46576 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:28:41.308864 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.304002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.6.236.189:41105)
0925 01:28:41.304197 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.304203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.304259 (+ 56us) server_negotiation.cc:407] Connection header received
0925 01:28:41.304487 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.304492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.304580 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.304651 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.305030 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.305869 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.307266 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.307734 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.307839 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.308205 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.308319 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.308390 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.308521 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.308598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 01:28:41.326442 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.324483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.6.236.188:42307)
0925 01:28:41.324670 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.324676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.325159 (+ 483us) server_negotiation.cc:407] Connection header received
0925 01:28:41.325325 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.325331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.325415 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.325501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.326203 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46588: BlockingRecv error: recv got EOF from 127.0.0.1:46588 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:28:41.336345 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.331254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.6.236.189:41105)
0925 01:28:41.331442 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.331448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.331904 (+ 456us) server_negotiation.cc:407] Connection header received
0925 01:28:41.332071 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.332075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.332142 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.332220 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.332614 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.333366 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.334679 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.335118 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.335225 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.335606 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.335721 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.335792 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.335939 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.336031 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:28:41.352131 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.343880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.6.236.188:42307)
0925 01:28:41.344059 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.344065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.350762 (+ 6697us) server_negotiation.cc:407] Connection header received
0925 01:28:41.350940 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.350944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.351016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.351122 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.351884 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46596: BlockingRecv error: recv got EOF from 127.0.0.1:46596 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:28:41.363695 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.358542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.6.236.189:41105)
0925 01:28:41.358789 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.358795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.359233 (+ 438us) server_negotiation.cc:407] Connection header received
0925 01:28:41.359412 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.359417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.359482 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.359551 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.359934 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.360723 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.362022 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.362531 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.362640 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.363022 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.363132 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.363214 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.363344 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.363429 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260925 01:28:41.376981 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.374862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.6.236.188:42307)
0925 01:28:41.375163 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.375169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.375689 (+ 520us) server_negotiation.cc:407] Connection header received
0925 01:28:41.375871 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.375877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.375953 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.376035 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.376771 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
I20260925 01:28:41.393654 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.387940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.6.236.189:41105)
0925 01:28:41.388154 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.388161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.388641 (+ 480us) server_negotiation.cc:407] Connection header received
0925 01:28:41.388888 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.388894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.388983 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.389046 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.389580 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.390414 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.391819 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.392343 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.392449 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.392848 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.392995 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.393077 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.393236 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.393331 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:28:41.401049 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.399544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.6.236.188:42307)
0925 01:28:41.399711 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.399717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.399737 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:28:41.399910 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.399916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.399998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.400055 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.400875 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46626: BlockingRecv error: recv got EOF from 127.0.0.1:46626 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:28:41.426039 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.420701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.6.236.189:41105)
0925 01:28:41.420941 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.420948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.421425 (+ 477us) server_negotiation.cc:407] Connection header received
0925 01:28:41.421620 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.421625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.421692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.421771 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.422176 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.423029 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.424351 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.424800 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.424913 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.425342 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.425449 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.425522 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.425663 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.425749 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 01:28:41.437393 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.435342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.6.236.188:42307)
0925 01:28:41.435549 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.435555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.436053 (+ 498us) server_negotiation.cc:407] Connection header received
0925 01:28:41.436233 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.436238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.436316 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.436400 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.437161 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46640: BlockingRecv error: recv got EOF from 127.0.0.1:46640 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:28:41.478472 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.472761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.6.236.189:41105)
0925 01:28:41.473013 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.473020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.473540 (+ 520us) server_negotiation.cc:407] Connection header received
0925 01:28:41.473752 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.473756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.473850 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.473928 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.474404 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.475161 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.476541 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.477055 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.477130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.477630 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.477742 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.477849 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.477980 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.478067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 01:28:41.486173 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.484568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.6.236.188:42307)
0925 01:28:41.484750 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.484755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.484774 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:28:41.484921 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.484927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.485012 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.485069 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.485984 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46648: BlockingRecv error: recv got EOF from 127.0.0.1:46648 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:28:41.561244 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.555718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.6.236.189:41105)
0925 01:28:41.555922 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.555930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.556435 (+ 505us) server_negotiation.cc:407] Connection header received
0925 01:28:41.556638 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.556643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.556713 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.556794 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.557239 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.558029 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.559458 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.559964 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.560043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.560531 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.560640 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.560718 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.560868 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.560951 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:28:41.570875 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.568626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.6.236.188:42307)
0925 01:28:41.568837 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.568844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.569428 (+ 584us) server_negotiation.cc:407] Connection header received
0925 01:28:41.569609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.569615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.569692 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.569779 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.570646 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46660: BlockingRecv error: recv got EOF from 127.0.0.1:46660 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:28:41.706657 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.702004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.6.236.189:41105)
0925 01:28:41.702303 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.702310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.702502 (+ 192us) server_negotiation.cc:407] Connection header received
0925 01:28:41.702672 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.702676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.702759 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.702839 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.703170 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.703870 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.705111 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.705525 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.705613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.705968 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.706089 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.706148 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.706321 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.706385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260925 01:28:41.714485 30098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:28:41.712689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.6.236.188:42307)
0925 01:28:41.712888 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.712894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.713138 (+ 244us) server_negotiation.cc:407] Connection header received
0925 01:28:41.713326 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.713330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.713414 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.713496 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:41.714286 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46668: BlockingRecv error: recv got EOF from 127.0.0.1:46668 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:28:41.761686 30114 catalog_manager.cc:1259] Loaded cluster ID: 45838dcf266a4e128fdec8db46ed5744
I20260925 01:28:41.761782 30114 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: loading cluster ID for follower catalog manager: success
I20260925 01:28:41.764349 30114 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: acquiring CA information for follower catalog manager: success
I20260925 01:28:41.766058 30114 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c03ec591990a400387a1aad21b889767: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:28:41.980099 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.975269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.6.236.189:41105)
0925 01:28:41.975487 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.975494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.975658 (+ 164us) server_negotiation.cc:407] Connection header received
0925 01:28:41.975829 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.975833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.975919 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.975998 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.976387 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.977125 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.978518 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.978925 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.979032 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.979423 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.979533 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.979598 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.979777 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.979842 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:28:41.991094 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.985694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.6.236.188:42307)
0925 01:28:41.985894 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.985900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.986045 (+ 145us) server_negotiation.cc:407] Connection header received
0925 01:28:41.986228 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.986232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.986415 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.986532 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.987061 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.987737 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.988960 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:41.989395 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:41.989485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:41.989932 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:41.990612 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:41.990731 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:41.990814 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 01:28:41.990883 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:28:42.004375 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.996481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.6.236.189:41105)
0925 01:28:41.996670 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.996678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.998771 (+ 2093us) server_negotiation.cc:407] Connection header received
0925 01:28:41.998946 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.998951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.999040 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:41.999119 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:41.999669 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.000680 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.002664 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.003061 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.003169 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:42.003648 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:42.003780 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:42.003865 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:42.003989 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 01:28:42.004068 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:28:42.005889 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.996423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.6.236.190:33291)
0925 01:28:41.999213 (+ 2790us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.999218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:41.999236 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:28:41.999340 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:41.999344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:41.999493 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:42.000356 (+ 863us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:42.000890 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.001740 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.003747 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.004300 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.004420 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:42.004916 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:42.005271 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:42.005367 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:42.005501 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 01:28:42.005583 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2675,"thread_start_us":2569,"threads_started":1}
I20260925 01:28:42.006572 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:41.997554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.6.236.188:42307)
0925 01:28:41.997697 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:41.997702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:42.000143 (+ 2441us) server_negotiation.cc:407] Connection header received
0925 01:28:42.000282 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:42.000286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:42.000361 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:42.000446 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:28:42.001225 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.002333 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.004286 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:42.004886 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:42.004968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:42.005595 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:28:42.005741 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:28:42.005792 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:42.006208 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 01:28:42.006336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:28:42.010694 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29880
I20260925 01:28:42.011569 29942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:42.119434 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29880
W20260925 01:28:42.137637 29973 connection.cc:535] server connection from 127.0.0.1:53914 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:28:42.138177 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 29954
I20260925 01:28:42.139142 30016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:42.244449 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 29954
I20260925 01:28:42.261745 7090 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgybJNj/build/asan/bin/kudu pid 30028
I20260925 01:28:42.262827 30090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:28:42.368237 7090 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgybJNj/build/asan/bin/kudu with pid 30028
2026-09-25T01:28:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4413 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8797 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346957 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST